Complaint detection method, device and equipment based on multi-modal model and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the field of financial science and technology, and discloses a complaint detection method, device and equipment based on a multi-modal model and a storage medium. Determining a first training data set and a second training data set; obtaining a first loss value of the multi-modal model on the first training data set, and obtaining a second loss value of the multi-modal model on the second training data set; when the first loss value or the second loss value meets a preset condition, obtaining a predicted complaint detection result output by the multi-modal model based on preset evaluation data; training a multi-modal model based on the difference between the predicted complaint detection result and the actual complaint detection result; and obtaining a current complaint detection result output by the trained multi-modal model based on the current evaluation data. The complaint detection efficiency of the current evaluation data can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology and can be applied to the field of financial technology. In particular, the present invention relates to a complaint detection method, device, equipment and storage medium based on a multimodal model. Background Art

[0002] As the influence of evaluation data on purchasing decisions continues to grow, complaint detection of the current evaluation data of the target product can obtain specific problems reported by users. Improving the target product through specific problems can help companies respond quickly and solve user pain points, thereby improving user experience.

[0003] However, the current complaint detection process of evaluation data is cumbersome, which is not conducive to improving the complaint detection efficiency of the current complaint detection results. The reason is that the existing technology relies on manual operation to perform complaint detection on the current evaluation data, and the manual operation consumes human resources and time resources, thus increasing the complaint detection time of the current evaluation data, and therefore, is not conducive to improving the complaint detection efficiency of the current evaluation data. Summary of the invention

[0004] The present invention provides a complaint detection method, device, computer equipment and storage medium based on a multimodal model to solve the technical problem that the complaint detection process of current evaluation data is cumbersome and is not conducive to improving the complaint detection efficiency of the current complaint detection results.

[0005] In a first aspect, a complaint detection method based on a multimodal model is provided, comprising:

[0006] Obtaining preset evaluation data of the target product and actual complaint test results corresponding to the preset evaluation data;

[0007] Determining a first training data set and a second training data set based on preset audio data, preset video data, and preset text data in the preset evaluation data;

[0008] Obtaining a first loss value of a preset multimodal model on the first training data set, and obtaining a second loss value of the multimodal model on the second training data set;

[0009] When the first loss value or the second loss value meets a preset condition, obtaining a predicted complaint detection result output by the multimodal model based on the preset evaluation data;

[0010] Based on the difference between the predicted complaint detection result and the actual complaint detection result, training the multimodal model, and determining the multimodal model after training;

[0011] Acquire current evaluation data of the target product, and acquire current complaint detection results output by the trained multimodal model based on the current evaluation data.

[0012] Furthermore, the determining of the first training data set and the second training data set based on the preset audio data, the preset video data, and the preset text data in the preset evaluation data includes:

[0013] Based on the preset text data, obtain a plurality of positive sample pairs corresponding to the preset audio data and a plurality of negative sample pairs corresponding to the preset audio data, and obtain a plurality of positive sample pairs corresponding to the preset video data and a plurality of negative sample pairs corresponding to the preset video data;

[0014] A first training data set is formed based on multiple positive sample pairs corresponding to the preset audio data and multiple negative sample pairs corresponding to the preset audio data, and a second training data set is formed based on multiple positive sample pairs corresponding to the preset video data and multiple negative sample pairs corresponding to the preset video data.

[0015] Further, the obtaining of a first loss value of a preset multimodal model on the first training data set and the obtaining of a second loss value of the multimodal model on the second training data set include:

[0016] Inputting the first training data set into a preset multimodal model, and using a preset contrast loss function to obtain a first loss value of the preset multimodal model on the first training data set;

[0017] The second training data set is input into the multimodal model, and the contrast loss function is used to obtain a second loss value of the multimodal model in the second training data set.

[0018] Furthermore, when the first loss value or the second loss value satisfies a preset condition, obtaining a predicted complaint detection result output by the multimodal model based on the preset evaluation data includes:

[0019] When the first loss value or the second loss value is less than a first preset value, feature extraction is performed on the preset audio data to obtain a first audio feature, feature extraction is performed on the preset video data to obtain a first video feature, and feature extraction is performed on the preset text data to obtain a first text feature;

[0020] The first audio feature, the first video feature, and the first text feature are fused to obtain a first target feature, and the first target feature is input into the multimodal model to obtain a predicted complaint detection result output by the multimodal model based on the first target feature.

[0021] Further, the training of the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and determining the trained multimodal model, comprises:

[0022] Obtain the predicted complaint status identifier and the predicted complaint category identifier in the predicted complaint detection result, obtain the actual complaint status identifier and the actual complaint category identifier in the actual complaint detection result, obtain the third loss value between the predicted complaint status identifier and the actual complaint status identifier, and obtain the fourth loss value between the predicted complaint category identifier and the actual complaint category identifier;

[0023] The first loss value, the second loss value, the third loss value, and the fourth loss value are added to generate a total loss value, and the multimodal model is trained based on the total loss value. When the total loss value is less than a second preset value, the training of the multimodal model is stopped, and the trained multimodal model is saved.

[0024] Furthermore, the obtaining of the current evaluation data of the target product and the obtaining of the current complaint detection result output by the trained multimodal model based on the current evaluation data include:

[0025] Acquire current evaluation data of the target product, perform feature extraction on current audio data in the current evaluation data to obtain a second audio feature, perform feature extraction on current video data in the current evaluation data to obtain a second video feature, and perform feature extraction on current text data in the current evaluation data to obtain a second text feature;

[0026] The second audio feature, the second video feature, and the second text feature are fused to obtain a second target feature, and the second target feature is input into the trained multimodal model to obtain a current complaint detection result output by the trained multimodal model based on the second target feature.

[0027] Further, after obtaining the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection method includes:

[0028] A display page corresponding to the target product is obtained, and the current complaint detection result is displayed through the display page. The current complaint detection result includes a current complaint status identifier and a current complaint category identifier.

[0029] In a second aspect, a complaint detection device based on a multimodal model is provided, comprising:

[0030] A first acquisition module is used to acquire preset evaluation data of a target product and actual complaint detection results corresponding to the preset evaluation data;

[0031] A first determination module, configured to determine a first training data set and a second training data set based on preset audio data, preset video data, and preset text data in the preset evaluation data;

[0032] A second acquisition module, used to acquire a first loss value of a preset multimodal model on the first training data set, and acquire a second loss value of the multimodal model on the second training data set;

[0033] A third acquisition module, configured to acquire a predicted complaint detection result output by the multimodal model based on the preset evaluation data when the first loss value or the second loss value meets a preset condition;

[0034] a second determination module, configured to train the multimodal model based on a difference between the predicted complaint detection result and the actual complaint detection result, and determine the multimodal model after training;

[0035] The detection module is used to obtain the current evaluation data of the target product and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data.

[0036] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above-mentioned complaint detection method are implemented when the processor executes the computer program.

[0037] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned complaint detection method are implemented.

[0038] The present application provides a complaint detection method, apparatus, computer equipment and storage medium based on a multimodal model, which has beneficial effects in two aspects. On the one hand, the current evaluation data of the target product is obtained, and the current complaint detection result output by the trained multimodal model based on the current evaluation data is obtained. Since no manual operation is required, the complaint detection time of the current evaluation data is reduced, which is beneficial to improving the complaint detection efficiency of the current evaluation data. On the other hand, since the multimodal model will not be affected by manual intervention, it is beneficial to improve the reliability of the current complaint detection result of the current evaluation data obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0040] Figure 1 is a schematic diagram of an application environment of a complaint detection method in one embodiment of the present invention;

[0041] Figure 2 A flowchart of a complaint detection method provided by an embodiment of the present invention;

[0042] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S23;

[0043] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S25;

[0044] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S26;

[0045] Figure 6 is a structural schematic diagram of a complaint detection device in one embodiment of the present invention;

[0046] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0047] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an application environment of a complaint detection method according to an embodiment of the present invention. The complaint detection method provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, a client communicates with a server through a network.

[0050] The server obtains preset evaluation data of the target product and the actual complaint detection results corresponding to the preset evaluation data through the client;

[0051] Determining a first training data set and a second training data set based on preset audio data, preset video data, and preset text data in the preset evaluation data;

[0052] Obtaining a first loss value of a preset multimodal model on the first training data set, and obtaining a second loss value of the multimodal model on the second training data set;

[0053] When the first loss value or the second loss value meets a preset condition, obtaining a predicted complaint detection result output by the multimodal model based on the preset evaluation data;

[0054] Based on the difference between the predicted complaint detection result and the actual complaint detection result, training the multimodal model, and determining the multimodal model after training;

[0055] Acquire current evaluation data of the target product, and acquire current complaint detection results output by the trained multimodal model based on the current evaluation data.

[0056] In the scheme implemented by the above-mentioned complaint detection method, device, equipment and medium, the beneficial effects are in two aspects. On the one hand, the current evaluation data of the target product is obtained, and the current complaint detection result output by the trained multimodal model based on the current evaluation data is obtained. Since no manual operation is required, the complaint detection time of the current evaluation data is reduced, which is beneficial to improving the complaint detection efficiency of the current evaluation data; on the other hand, since the multimodal model will not be affected by human intervention, it is beneficial to improve the reliability of the current complaint detection result of the current evaluation data obtained.

[0057] The device running the client is referred to as a client device.

[0058] The device running the server is referred to as the server device.

[0059] The client devices may include but are not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0060] The server device can be implemented with an independent task database or a task database cluster composed of multiple task databases. The present invention is described in detail below through specific embodiments.

[0061] See also Figure 2 , Figure 2A flowchart of a complaint detection method provided by an embodiment of the present invention includes the following steps:

[0062] S21, obtaining preset evaluation data of the target product and actual complaint detection results corresponding to the preset evaluation data;

[0063] Among them, the target products include one or a combination of insurance products, financial products, and technology products.

[0064] Insurance products are a combination of committed services provided by insurance companies to the market, aimed at meeting consumers' needs for risk reduction and risk transfer. Financial products include but are not limited to life insurance, health insurance, and property insurance.

[0065] Financial products refer to various financial instruments and services provided by financial institutions or financial markets to meet the investment, financing, risk management and asset allocation needs of individuals, enterprises and governments. Financial products include but are not limited to bank deposits, bonds, stocks, foreign exchange, etc.

[0066] Among them, technology products refer to various new products or equipment developed and manufactured using modern science and technology. Technology products include but are not limited to smart phones, tablet computers, wearable devices, etc.

[0067] S22, determining a first training data set and a second training data set based on the preset audio data, the preset video data, and the preset text data in the preset evaluation data;

[0068] Wherein, determining the first training data set and the second training data set based on the preset audio data, the preset video data, and the preset text data in the preset evaluation data includes:

[0069] Based on the preset text data, obtain a plurality of positive sample pairs corresponding to the preset audio data and a plurality of negative sample pairs corresponding to the preset audio data, and obtain a plurality of positive sample pairs corresponding to the preset video data and a plurality of negative sample pairs corresponding to the preset video data;

[0070] A first training data set is formed based on multiple positive sample pairs corresponding to the preset audio data and multiple negative sample pairs corresponding to the preset audio data, and a second training data set is formed based on multiple positive sample pairs corresponding to the preset video data and multiple negative sample pairs corresponding to the preset video data.

[0071] It should be noted that, based on the preset text data, obtaining a plurality of positive sample pairs corresponding to the preset audio data and a plurality of negative sample pairs corresponding to the preset audio data, and obtaining a plurality of positive sample pairs corresponding to the preset video data and a plurality of negative sample pairs corresponding to the preset video data include:

[0072] In the preset text data, a text description matching the preset audio data is selected as a first text description, a text description not matching the preset audio data is selected as a second text description, a text description matching the preset video data is selected as a third text description, and a text description not matching the preset video data is selected as a fourth text description;

[0073] Based on the preset audio data and the multiple first text descriptions, multiple positive sample pairs corresponding to the preset audio data are constructed; based on the preset audio data and the multiple second text descriptions, multiple negative sample pairs corresponding to the preset audio data are constructed; based on the multiple positive sample pairs corresponding to the preset audio data and the multiple negative sample pairs corresponding to the preset audio data, a first training data set is formed.

[0074] Based on the preset video data and the multiple third text descriptions, multiple positive sample pairs corresponding to the preset video data are constructed; based on the preset video data and the multiple fourth text descriptions, multiple negative sample pairs corresponding to the preset video data are constructed; based on the multiple positive sample pairs corresponding to the preset video data and the multiple negative sample pairs corresponding to the preset video data, a second training data set is formed.

[0075] Among them, the first loss value is an important indicator to measure the degree of alignment between the preset audio data and the first text description. By minimizing the first loss value, the multimodal model can learn a more accurate alignment method, making the preset audio data and the first text description closer in the feature space.

[0076] Among them, the second loss value is an important indicator to measure the degree of alignment between the preset video data and the third text description. By minimizing the second loss value, the multimodal model can learn a more accurate alignment method, making the preset video data and the third text description closer in the feature space.

[0077] Among them, obtaining the first loss value of the preset multimodal model on the first training data set can effectively promote the multimodal model to achieve alignment between the preset audio data and the first text description, and obtaining the second loss value of the multimodal model on the second training data set can effectively promote the multimodal model to achieve alignment between the preset video data and the third text description. This cross-modal alignment enables the multimodal model to better adapt to complex and changeable actual application scenarios.

[0078] S23, obtaining a first loss value of a preset multimodal model on the first training data set, and obtaining a second loss value of the multimodal model on the second training data set;

[0079] S24, when the first loss value or the second loss value meets a preset condition, obtaining a predicted complaint detection result output by the multimodal model based on the preset evaluation data;

[0080] Wherein, when the first loss value or the second loss value meets a preset condition, obtaining the predicted complaint detection result output by the multimodal model based on the preset evaluation data includes:

[0081] When the first loss value or the second loss value is less than a first preset value, feature extraction is performed on the preset audio data to obtain a first audio feature, feature extraction is performed on the preset video data to obtain a first video feature, and feature extraction is performed on the preset text data to obtain a first text feature;

[0082] The first audio feature, the first video feature, and the first text feature are fused to obtain a first target feature, and the first target feature is input into the multimodal model to obtain a predicted complaint detection result output by the multimodal model based on the first target feature.

[0083] S25, training the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and determining the multimodal model after training;

[0084] Among them, training the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result can optimize the prediction ability of the multimodal model, because the modal model can gradually narrow the difference by continuously comparing and correcting the difference between the predicted complaint detection result and the actual complaint detection result, thereby improving the accuracy and reliability of the multimodal model prediction.

[0085] S26, obtaining current evaluation data of the target product, and obtaining current complaint detection results output by the trained multimodal model based on the current evaluation data.

[0086] Among them, the target products include one or a combination of insurance products, financial products, and technology products.

[0087] For the sake of illustration, an example is given below:

[0088] For example, when the current evaluation data is the current video content and the target product is an insurance product, the insurance product has been promoted on various video platforms, and users have made various comments on the current video content of the insurance product, as well as corresponding complaints or negative news. This method can be applied in the video marketing scenario of insurance products to locate and detect user complaints about the current video content of the insurance product, thereby optimizing the insurance product and improving the service quality.

[0089] For example, when the current evaluation data is the current video content and the target product is a financial product, the financial product has been promoted on various video platforms, and users have made various comments on the current video content of the financial product, as well as corresponding complaints or negative news. This method can be applied in the video marketing scenario of financial products to locate and detect user complaints about the current video content of the financial product, thereby optimizing financial products and improving service quality.

[0090] For example, when the current evaluation data is the current video content and the target product is a technology product, the technology product has been promoted on various video platforms, and users have made various comments on the current video content of the technology product, as well as corresponding complaints or negative news. This method can be applied in the video marketing scenario of technology products to locate and detect user complaints about the current video content of technology products, thereby optimizing technology products and improving service quality.

[0091] Wherein, after obtaining the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection method includes:

[0092] A display page corresponding to the target product is obtained, and the current complaint detection result is displayed through the display page. The current complaint detection result includes a current complaint status identifier and a current complaint category identifier.

[0093] Among them, the current complaint detection result is the complaint detection result predicted by the trained multimodal model.

[0094] Among them, the current complaint status identifier is the complaint status identifier predicted by the trained multimodal model.

[0095] The complaint status is indicated as a complaint that has been filed or a non-complaint that has not been filed.

[0096] Among them, the current complaint category identifier is the complaint category identifier predicted by the trained multimodal model.

[0097] Among them, the complaint category identification includes but is not limited to the product quality category identification, the product description discrepancy category identification, the price issue category identification, the after-sales service category identification, and the pre-sales service category identification.

[0098] In the embodiments of the present invention, the beneficial effects are in two aspects. On the one hand, the current evaluation data of the target product is obtained, and the current complaint detection result output by the trained multimodal model based on the current evaluation data is obtained. Since no manual operation is required, the complaint detection time of the current evaluation data is reduced, which is beneficial to improving the complaint detection efficiency of the current evaluation data. On the other hand, since the multimodal model will not be affected by human intervention, it is beneficial to improve the reliability of the current complaint detection result of the current evaluation data obtained.

[0099] See also Figure 3 , Figure 3 yes Figure 1 A specific implementation flow diagram of step S23 is described in detail as follows:

[0100] S31, inputting the first training data set into a preset multimodal model, and using a preset contrast loss function to obtain a first loss value of the preset multimodal model on the first training data set;

[0101] S32: Input the second training data set into the multimodal model, and use the contrast loss function to obtain a second loss value of the multimodal model in the second training data set.

[0102] In the embodiment of the present invention, the first loss value and the second loss value are important indicators for measuring the degree of alignment between multimodal data. By minimizing the first loss value and the second loss value, the multimodal model can learn a more accurate alignment method, thereby improving the multimodal model's ability to understand and process multimodal data.

[0103] See also Figure 4 , Figure 4 yes Figure 1 A specific implementation flow diagram of step S25 is described in detail as follows:

[0104] S41, obtaining the predicted complaint status identifier and the predicted complaint category identifier in the predicted complaint detection result, obtaining the actual complaint status identifier and the actual complaint category identifier in the actual complaint detection result, obtaining the third loss value between the predicted complaint status identifier and the actual complaint status identifier, and obtaining the fourth loss value between the predicted complaint category identifier and the actual complaint category identifier;

[0105] Among them, the predicted complaint detection result is the complaint detection result predicted by the multimodal model whose training has not been completed.

[0106] Among them, the predicted complaint status identifier is the complaint status identifier predicted by the multimodal model whose training has not been completed.

[0107] Among them, the predicted complaint category identifier is the complaint category identifier predicted by the multimodal model whose training has not been completed.

[0108] Among them, the actual complaint test results are the marked complaint test results.

[0109] Among them, the actual complaint status identifier is the marked complaint status identifier.

[0110] Among them, the actual complaint category identifier is the marked complaint category identifier.

[0111] S42, adding the first loss value, the second loss value, the third loss value, and the fourth loss value to generate a total loss value, training the multimodal model based on the total loss value, and when the total loss value is less than a second preset value, stopping training the multimodal model and saving the trained multimodal model.

[0112] For the sake of illustration, an example is given below:

[0113] Based on a preset loss value generation model, the first loss value, the second loss value, the third loss value, and the fourth loss value are added to generate a total loss value. The loss value generation model is:

[0114]

[0115] in, represents the total loss value, represents the first loss value, represents the second loss value, represents the third loss value, represents the fourth loss value.

[0116] In an embodiment of the present invention, the trained multimodal model is saved, and the trained multimodal model can be deployed at any time without the need to re-perform the time-consuming and resource-intensive training process, thereby greatly improving work efficiency.

[0117] See also Figure 5 , Figure 5 yes Figure 1 A specific implementation flow diagram of step S26 is described in detail as follows:

[0118] S51, obtaining current evaluation data of the target product, performing feature extraction on current audio data in the current evaluation data to obtain a second audio feature, performing feature extraction on current video data in the current evaluation data to obtain a second video feature, and performing feature extraction on current text data in the current evaluation data to obtain a second text feature;

[0119] S52, fusing the second audio feature, the second video feature, and the second text feature to obtain a second target feature, inputting the second target feature into the trained multimodal model, and obtaining a current complaint detection result output by the trained multimodal model based on the second target feature.

[0120] In an embodiment of the present invention, the current complaint detection result output by the trained multimodal model based on the second target feature is obtained. Since no manual operation is required, the complaint detection time of the current evaluation data is reduced, which is beneficial to improving the complaint detection efficiency of the current evaluation data.

[0121] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of a complaint detection device according to an embodiment of the present invention. Figure 6 As shown, the complaint detection device includes a first acquisition module 101, a first determination module 102, a second acquisition module 103, a third acquisition module 104, a second determination module 105, and a detection module 106. The functional modules are described in detail as follows:

[0122] The first acquisition module 101 is used to acquire preset evaluation data of a target product and actual complaint detection results corresponding to the preset evaluation data;

[0123] A first determination module 102, configured to determine a first training data set and a second training data set based on the preset audio data, the preset video data, and the preset text data in the preset evaluation data;

[0124] A second acquisition module 103 is used to acquire a first loss value of a preset multimodal model on the first training data set, and acquire a second loss value of the multimodal model on the second training data set;

[0125] A third acquisition module 104 is used to acquire a predicted complaint detection result output by the multimodal model based on the preset evaluation data when the first loss value or the second loss value meets a preset condition;

[0126] A second determination module 105, configured to train the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and determine the multimodal model after training;

[0127] The detection module 106 is used to obtain the current evaluation data of the target product and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data.

[0128] In one embodiment, the first determining module 102 includes:

[0129] A first acquisition subunit is used to acquire, based on the preset text data, a plurality of positive sample pairs corresponding to the preset audio data and a plurality of negative sample pairs corresponding to the preset audio data, and to acquire a plurality of positive sample pairs corresponding to the preset video data and a plurality of negative sample pairs corresponding to the preset video data;

[0130] The composition subunit is used to form a first training data set based on a plurality of positive sample pairs corresponding to the preset audio data and a plurality of negative sample pairs corresponding to the preset audio data, and to form a second training data set based on a plurality of positive sample pairs corresponding to the preset video data and a plurality of negative sample pairs corresponding to the preset video data.

[0131] In one embodiment, the second acquisition module 103 includes:

[0132] a second acquisition subunit, configured to input the first training data set into a preset multimodal model, and use a preset contrast loss function to obtain a first loss value of the preset multimodal model on the first training data set;

[0133] The third acquisition subunit is used to input the second training data set into the multimodal model, and use the contrast loss function to obtain a second loss value of the multimodal model in the second training data set.

[0134] In one embodiment, the third acquisition module 104 includes:

[0135] A first extraction subunit is configured to perform feature extraction on the preset audio data to obtain a first audio feature, perform feature extraction on the preset video data to obtain a first video feature, and perform feature extraction on the preset text data to obtain a first text feature when the first loss value or the second loss value is less than a first preset value;

[0136] An input subunit is used to fuse the first audio feature, the first video feature, and the first text feature to obtain a first target feature, input the first target feature into the multimodal model, and obtain a predicted complaint detection result output by the multimodal model based on the first target feature.

[0137] In one embodiment, the second determining module 105 includes:

[0138] A fourth acquisition subunit is used to obtain the predicted complaint status identifier and the predicted complaint category identifier in the predicted complaint detection result, obtain the actual complaint status identifier and the actual complaint category identifier in the actual complaint detection result, obtain the third loss value between the predicted complaint status identifier and the actual complaint status identifier, and obtain the fourth loss value between the predicted complaint category identifier and the actual complaint category identifier;

[0139] A generation subunit is used to add the first loss value, the second loss value, the third loss value, and the fourth loss value to generate a total loss value, and train the multimodal model based on the total loss value. When the total loss value is less than a second preset value, the training of the multimodal model is stopped, and the trained multimodal model is saved.

[0140] In one embodiment, the detection module 106 includes:

[0141] A second extraction subunit is used to obtain the current evaluation data of the target product, perform feature extraction on the current audio data in the current evaluation data to obtain a second audio feature, perform feature extraction on the current video data in the current evaluation data to obtain a second video feature, and perform feature extraction on the current text data in the current evaluation data to obtain a second text feature;

[0142] The detection subunit is used to fuse the second audio feature, the second video feature, and the second text feature to obtain a second target feature, input the second target feature into the trained multimodal model, and obtain the current complaint detection result output by the trained multimodal model based on the second target feature.

[0143] In one embodiment, the complaint detection device further includes:

[0144] A display module is used to obtain a display page corresponding to the target product, and display the current complaint detection result through the display page. The current complaint detection result includes a current complaint status identifier and a current complaint category identifier.

[0145] In the embodiments of the present invention, the beneficial effects are in two aspects. On the one hand, the current evaluation data of the target product is obtained, and the current complaint detection result output by the trained multimodal model based on the current evaluation data is obtained. Since no manual operation is required, the complaint detection time of the current evaluation data is reduced, which is beneficial to improving the complaint detection efficiency of the current evaluation data. On the other hand, since the multimodal model will not be affected by human intervention, it is beneficial to improve the reliability of the current complaint detection result of the current evaluation data obtained.

[0146] For the specific limitations on the complaint detection device, please refer to the limitations on the complaint detection method above, which will not be repeated here.

[0147] Each module in the above complaint detection device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0148] See also Figure 7 , Figure 7 is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a server device, and its internal structure diagram may be as shown in FIG. Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus.

[0149] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, the functions or steps of a complaint detection method based on a multimodal model can be implemented on the server-side device.

[0150] See also Figure 8 , Figure 8 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a client device, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external task database via a network connection. When the computer program is executed by the processor, the functions or steps of a complaint detection method based on a multimodal model can be implemented on a client device.

[0151] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor.

[0152] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.

[0153] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0154] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present disclosure.

Claims

1. A complaint detection method based on a multimodal model, characterized in that: include: Obtaining preset evaluation data of the target product and actual complaint test results corresponding to the preset evaluation data; Determining a first training data set and a second training data set based on preset audio data, preset video data, and preset text data in the preset evaluation data; Obtaining a first loss value of a preset multimodal model on the first training data set, and obtaining a second loss value of the multimodal model on the second training data set; When the first loss value or the second loss value meets a preset condition, obtaining a predicted complaint detection result output by the multimodal model based on the preset evaluation data; Based on the difference between the predicted complaint detection result and the actual complaint detection result, the multimodal model is trained to determine the trained multimodal model; the current evaluation data of the target product is obtained, and the current complaint detection result output by the trained multimodal model based on the current evaluation data is obtained.

2. The complaint detection method according to claim 1, characterized in that: The determining of the first training data set and the second training data set based on the preset audio data, the preset video data, and the preset text data in the preset evaluation data includes: Based on the preset text data, obtain a plurality of positive sample pairs corresponding to the preset audio data and a plurality of negative sample pairs corresponding to the preset audio data, and obtain a plurality of positive sample pairs corresponding to the preset video data and a plurality of negative sample pairs corresponding to the preset video data; A first training data set is formed based on multiple positive sample pairs corresponding to the preset audio data and multiple negative sample pairs corresponding to the preset audio data, and a second training data set is formed based on multiple positive sample pairs corresponding to the preset video data and multiple negative sample pairs corresponding to the preset video data.

3. The complaint detection method according to claim 1, characterized in that: The obtaining of a first loss value of a preset multimodal model on the first training data set and the obtaining of a second loss value of the multimodal model on the second training data set include: Inputting the first training data set into a preset multimodal model, and using a preset contrast loss function to obtain a first loss value of the preset multimodal model on the first training data set; The second training data set is input into the multimodal model, and the contrast loss function is used to obtain a second loss value of the multimodal model in the second training data set.

4. The complaint detection method according to claim 1, characterized in that: When the first loss value or the second loss value satisfies a preset condition, obtaining a predicted complaint detection result output by the multimodal model based on the preset evaluation data includes: When the first loss value or the second loss value is less than a first preset value, feature extraction is performed on the preset audio data to obtain a first audio feature, feature extraction is performed on the preset video data to obtain a first video feature, and feature extraction is performed on the preset text data to obtain a first text feature; The first audio feature, the first video feature, and the first text feature are fused to obtain a first target feature, and the first target feature is input into the multimodal model to obtain a predicted complaint detection result output by the multimodal model based on the first target feature.

5. The complaint detection method according to claim 1, characterized in that: The step of training the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and determining the multimodal model after training, comprises: Obtain the predicted complaint status identifier and the predicted complaint category identifier in the predicted complaint detection result, obtain the actual complaint status identifier and the actual complaint category identifier in the actual complaint detection result, obtain the third loss value between the predicted complaint status identifier and the actual complaint status identifier, and obtain the fourth loss value between the predicted complaint category identifier and the actual complaint category identifier; The first loss value, the second loss value, the third loss value, and the fourth loss value are added to generate a total loss value, and the multimodal model is trained based on the total loss value. When the total loss value is less than a second preset value, the training of the multimodal model is stopped, and the trained multimodal model is saved.

6. The complaint detection method according to claim 1, characterized in that: The obtaining of the current evaluation data of the target product and the obtaining of the current complaint detection result output by the trained multimodal model based on the current evaluation data include: Acquire current evaluation data of the target product, perform feature extraction on current audio data in the current evaluation data to obtain a second audio feature, perform feature extraction on current video data in the current evaluation data to obtain a second video feature, and perform feature extraction on current text data in the current evaluation data to obtain a second text feature; The second audio feature, the second video feature, and the second text feature are fused to obtain a second target feature, and the second target feature is input into the trained multimodal model to obtain a current complaint detection result output by the trained multimodal model based on the second target feature.

7. The complaint detection method according to any one of claims 1 to 6, characterized in that: After obtaining the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection method includes: A display page corresponding to the target product is obtained, and the current complaint detection result is displayed through the display page. The current complaint detection result includes a current complaint status identifier and a current complaint category identifier.

8. A complaint detection device based on a multimodal model, characterized in that: include: A first acquisition module is used to acquire preset evaluation data of a target product and actual complaint detection results corresponding to the preset evaluation data; A first determination module, configured to determine a first training data set and a second training data set based on preset audio data, preset video data, and preset text data in the preset evaluation data; A second acquisition module, used to acquire a first loss value of a preset multimodal model on the first training data set, and acquire a second loss value of the multimodal model on the second training data set; A third acquisition module, configured to acquire a predicted complaint detection result output by the multimodal model based on the preset evaluation data when the first loss value or the second loss value meets a preset condition; a second determination module, configured to train the multimodal model based on a difference between the predicted complaint detection result and the actual complaint detection result, and determine the multimodal model after training; The detection module is used to obtain the current evaluation data of the target product and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the complaint detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the complaint detection method according to any one of claims 1 to 7 are implemented.

Images