Abnormal information processing method and device, equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence and financial science and technology, and discloses an abnormal information processing method, device and equipment and a storage medium, and the method comprises the steps: obtaining abnormal information sent by user equipment, and the abnormal information comprises a screen image and description characters when an application program of the user equipment is abnormal; performing feature extraction on the screen image to obtain a first feature vector, and performing feature extraction on the description character to obtain a second feature vector; fusing the first feature vector and the second feature vector to generate a third feature vector; based on a preset multi-modal large model and the third feature vector, determining a query statement; encoding the query statement to generate a sentence vector; and in a preset vector database, querying a processing scheme of the abnormal information through the sentence vector, and sending the processing scheme to the user equipment. The processing efficiency of the abnormal information can be improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and financial technology, and in particular to methods, devices, equipment and storage media for processing abnormal information. Background Art

[0002] When using an application, users may encounter various problems, such as network anomalies, inability to jump, etc. Usually, users will take screenshots of the abnormal information of the application and feedback the screen image to the intelligent customer service of the application.

[0003] Since traditional intelligent customer service based on natural language processing cannot understand screen images, it cannot provide solutions for abnormal information and can only be handled by human intervention. However, human intervention has the following problems:

[0004] 1. High labor costs: Manual processing requires hiring and training enough customer service representatives to handle a large amount of exception information, which will lead to high labor costs, especially in high-traffic applications.

[0005] 2. Human limitation: The number of customer service representatives is limited, and they cannot handle a large number of abnormal information at the same time. This may lead to long waiting times and processing delays, thus reducing the user experience.

[0006] 3. Timeliness issue: Manual processing is limited by working hours and cannot provide 24 / 7 support. Users may not be able to get immediate help when they encounter problems during non-working hours or holidays.

[0007] In summary, the existing abnormal information processing process is cumbersome, which is not conducive to improving the abnormal information processing efficiency. Summary of the invention

[0008] The present invention provides an abnormal information processing method, device, computer equipment and storage medium to solve the technical problem that the existing abnormal information processing process is complicated and is not conducive to improving the abnormal information processing efficiency.

[0009] In a first aspect, a method for processing abnormal information is provided, comprising:

[0010] Acquire abnormal information sent by the user device, wherein the abnormal information includes a screen image and description text when an application of the user device has an abnormality;

[0011] Performing feature extraction on the screen image to obtain a first feature vector, and performing feature extraction on the caption text to obtain a second feature vector;

[0012] Based on a preset fusion method, the first feature vector and the second feature vector are fused to generate a third feature vector;

[0013] Determine a query statement based on a preset multimodal macro model and the third feature vector;

[0014] Encode the query sentence to generate a sentence vector;

[0015] Based on a preset query method, in a preset vector database, a processing solution for the abnormal information is queried through the sentence vector, and the processing solution is sent to the user equipment.

[0016] Further, the extracting features of the screen image to obtain a first feature vector and extracting features of the caption to obtain a second feature vector include:

[0017] Using a preset image feature extraction network to extract features from the screen image to obtain a first feature vector;

[0018] A preset text feature extraction network is used to extract features from the description text to obtain a second feature vector.

[0019] Furthermore, the step of fusing the first feature vector and the second feature vector based on a preset fusion method to generate a third feature vector includes:

[0020] Processing the first feature vector and the second feature vector through a preset attention mechanism;

[0021] The first feature vector and the second feature vector processed by the attention mechanism are concatenated to generate a third feature vector.

[0022] Furthermore, the determining of the query statement based on the preset multimodal macro model and the third feature vector includes:

[0023] Inputting the third feature vector into a preset multimodal macro model;

[0024] Obtain a query statement generated by the multimodal large model based on the third feature vector.

[0025] Furthermore, encoding the query sentence to generate a sentence vector includes:

[0026] Encode each word of the query sentence through a preset word embedding model to generate a word vector corresponding to each word;

[0027] The word vectors corresponding to each of the words are combined to generate the sentence vector.

[0028] Further, the querying of the processing scheme of the abnormal information in a preset vector database by using the sentence vector based on a preset query method, and sending the processing scheme to the user equipment includes:

[0029] Packing the sentence vector and the query instruction into a query request, and sending the query request to a preset vector database;

[0030] Receive the query result returned by the vector database after executing the query request, select the query result as the processing solution for the abnormal information, and send the processing solution to the user device, wherein the processing solution includes one of a text solution, an image solution, and a video solution, or a combination thereof.

[0031] Further, in the preset query mode, in a preset vector database, the processing scheme of the abnormal information is queried through the sentence vector, and after the processing scheme is sent to the user equipment, the abnormal information processing method includes:

[0032] A preset storage area is obtained, and the abnormal information and the processing solution are stored in the storage area.

[0033] In a second aspect, an abnormal information processing device is provided, comprising:

[0034] An acquisition module, used to acquire abnormal information sent by a user device, wherein the abnormal information includes a screen image and description text when an abnormality occurs in an application of the user device;

[0035] An extraction module, used for performing feature extraction on the screen image to obtain a first feature vector, and performing feature extraction on the caption text to obtain a second feature vector;

[0036] A first generating module, configured to fuse the first feature vector and the second feature vector based on a preset fusion method to generate a third feature vector;

[0037] A determination module, configured to determine a query statement based on a preset multimodal macro model and the third feature vector;

[0038] A second generation module is used to encode the query sentence and generate a sentence vector;

[0039] The processing module is used to query the processing solution of the abnormal information through the sentence vector in a preset vector database based on a preset query method, and send the processing solution to the user equipment.

[0040] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned exception information processing method when executing the computer program.

[0041] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned exception information processing method are implemented.

[0042] The present application provides an abnormal information processing method, device, computer equipment and storage medium, which obtain abnormal information sent by a user device, wherein the abnormal information includes a screen image and an explanatory text when an application of the user device is abnormal; feature extraction is performed on the screen image to obtain a first feature vector, and feature extraction is performed on the explanatory text to obtain a second feature vector; based on a preset fusion method, the first feature vector and the second feature vector are fused to generate a third feature vector; based on a preset multimodal large model and the third feature vector, a query statement is determined; the query statement is encoded to generate a sentence vector; based on a preset query method, a processing solution for the abnormal information is queried in a preset vector database through the sentence vector, and the processing solution is sent to the user device, and the beneficial effects are in two aspects. On the one hand, based on the preset query method, a processing solution for the abnormal information is queried in a vector database through the sentence vector, and the processing solution is sent to the user device. Since manual processing is not required, the processing time of the abnormal information is reduced, which is conducive to improving the processing efficiency of the abnormal information; on the other hand, since the vector database can process query requests in parallel, the query time of the processing solution for the abnormal information is reduced, which is conducive to further improving the processing efficiency of the abnormal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0044] Figure 1 is a schematic diagram of an application environment of an abnormal information processing method in one embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a flow chart of an abnormal information processing method provided by an embodiment of the present invention;

[0046] Figure 3 yes Figure 1A schematic flow chart of a specific implementation of step S23;

[0047] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S25;

[0048] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S26;

[0049] Figure 6 is a schematic diagram of the structure of an abnormal information processing device in one embodiment of the present invention;

[0050] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0051] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] See also Figure 1 , Figure 1 is a schematic diagram of an application environment of an abnormal information processing method according to an embodiment of the present invention. The abnormal information processing method provided by the embodiment of the present invention can be applied in the following embodiments: Figure 1 In an application environment, a user device communicates with a server through a network.

[0054] The server obtains the abnormal information sent by the user device, wherein the abnormal information includes a screen image and description text when the application of the user device is abnormal;

[0055] Performing feature extraction on the screen image to obtain a first feature vector, and performing feature extraction on the caption text to obtain a second feature vector;

[0056] Based on a preset fusion method, the first feature vector and the second feature vector are fused to generate a third feature vector;

[0057] Determine a query statement based on a preset multimodal macro model and the third feature vector;

[0058] Encode the query sentence to generate a sentence vector;

[0059] Based on a preset query method, in a preset vector database, a processing solution for the abnormal information is queried through the sentence vector, and the processing solution is sent to the user equipment.

[0060] Among them, the solution implemented by the above-mentioned abnormal information processing method, device, equipment and medium has the following advantages:

[0061] (1) Timeliness: The server is online at any time, without rest and without restrictions on working hours. It can provide 7*24 hours support, so user devices can submit abnormal information at any time and get immediate response and processing solutions. This provides faster support, especially during non-working hours or in emergency situations.

[0062] (2) Reduce the pressure on manual customer service: The server can handle common and repetitive exceptions, reducing the workload of the manual customer service team, allowing them to focus more on more complex issues and personalized support.

[0063] (3) Real-time update of the vector database: The knowledge in the vector database can be easily updated to reflect new abnormal situations and treatment solutions. This ensures that the server can always provide the latest recommendations.

[0064] (4) Cost savings: Servers can reduce the operating costs of an enterprise because there is no need to hire a large number of customer service staff, especially when dealing with repetitive issues.

[0065] (5) Improve user experience: User devices can more easily obtain immediate, consistent, and accurate support, which improves user experience and increases user satisfaction.

[0066] The user devices may include but are not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0067] The server can be implemented with an independent task database or a task database cluster composed of multiple task databases. The present invention is described in detail below through specific embodiments.

[0068] See also Figure 2 , Figure 2 A flowchart of an abnormal information processing method provided by an embodiment of the present invention includes the following steps:

[0069] S21, obtaining abnormal information sent by the user device, wherein the abnormal information includes a screen image and description text when an abnormality occurs in an application of the user device;

[0070] Application exceptions refer to situations that the program encounters unexpected situations during operation, which prevents the program from continuing to execute or completing its intended tasks. This is usually caused by a variety of factors, such as errors within the program, changes in the external environment, resource limitations, or improper user input.

[0071] The application includes one or a combination of insurance application, financial application, and technology application.

[0072] For the sake of explanation, the insurance procedure is taken as an example and described in detail as follows:

[0073] Insurance procedures refer to a series of business processes designed and implemented by insurance companies to manage risks and provide protection services. These procedures cover every aspect from the design, pricing, and sales of insurance products to underwriting, insurance coverage, and claims settlement. Specifically, insurance procedures include:

[0074] Product design: Design insurance products that meet customer needs based on market demand and risk assessment.

[0075] Sales and Insurance: Promote insurance products to customers through various channels and assist customers in completing insurance procedures.

[0076] Underwriting and insurance: Review the policyholder’s application, assess the risk and decide whether to provide insurance and determine the premium rate.

[0077] Claims service: After an insurance accident occurs, we provide customers with timely claims services according to the contract agreement.

[0078] The efficient operation of insurance procedures is of great significance to protecting customer rights and interests, maintaining market order and promoting the healthy development of the insurance industry.

[0079] For the sake of explanation, we take the financial procedure as an example and describe it in detail as follows:

[0080] Financial procedures refer to a series of business processes and technical means adopted by financial institutions to achieve financial functions such as financing, risk management, payment and settlement.

[0081] Financial procedures are widely involved in many fields such as banking, securities, insurance, and funds, and are an important part of the modern financial system. The main characteristics of financial procedures include:

[0082] Fund liquidity management: Through financial markets and financial instruments, funds can be raised, allocated and invested to improve the efficiency of fund use.

[0083] Risk management: Use various financial tools and strategies to effectively manage credit risk, market risk, operational risk, etc.

[0084] Payment and settlement: Provide safe and efficient payment and settlement services to ensure the smooth progress of economic activities.

[0085] The complexity and professionalism of financial procedures require financial institutions to have a high level of professionalism and risk management capabilities.

[0086] For the sake of explanation, we take the scientific and technological process as an example and describe it in detail as follows:

[0087] Technology programs refer to a series of instructions and algorithms that use technology to solve practical problems or achieve specific functions. In the insurance and financial fields, technology programs are increasingly used, such as:

[0088] Smart underwriting: Use big data and artificial intelligence technology to quickly and accurately review policyholders’ applications.

[0089] Automated claims: Through technologies such as image recognition and natural language processing, the claims process can be automated to improve claims efficiency.

[0090] The application of technological programs has not only improved the efficiency and quality of insurance and financial services, but also promoted innovation and change in the industry.

[0091] S22, extracting features from the screen image to obtain a first feature vector, and extracting features from the caption text to obtain a second feature vector;

[0092] The step of extracting features from the screen image to obtain a first feature vector and extracting features from the caption text to obtain a second feature vector includes:

[0093] Using a preset image feature extraction network to extract features from the screen image to obtain a first feature vector;

[0094] A preset text feature extraction network is used to extract features from the description text to obtain a second feature vector.

[0095] The screen image is input into the image feature extraction network to obtain the first feature vector output by the image feature extraction network.

[0096] Among them, the image feature extraction network can extract the first feature vector from the screen image after multiple layers of nonlinear transformation.

[0097] Among them, the image feature extraction network includes but is not limited to feature pyramid network and residual network.

[0098] The description text is input into a text feature extraction network to obtain a second feature vector output by the text feature extraction network.

[0099] Among them, the text feature extraction network includes but is not limited to convolutional neural network and convolutional recurrent neural network.

[0100] Among them, the text feature extraction network automatically extracts key information from the caption text, and this key information is represented in the form of a second feature vector. The second feature vector can reduce the complexity of the data and improve processing efficiency.

[0101] S23, based on a preset fusion method, fusing the first feature vector and the second feature vector to generate a third feature vector;

[0102] S24, determining a query statement based on a preset multimodal macro model and the third feature vector;

[0103] The step of determining the query statement based on the preset multimodal macro model and the third feature vector includes:

[0104] Inputting the third feature vector into a preset multimodal macro model;

[0105] Obtain a query statement generated by the multimodal large model based on the third feature vector.

[0106] The multimodal big model captures and analyzes information in different modal data from multiple dimensions by building a complex neural network structure and using deep learning technology. The multimodal big model can not only process data of each modality separately, but also learn the association and interaction between different modalities, thereby achieving a comprehensive understanding and application of multimodal data.

[0107] The multimodal large model includes one of the Bert model and the ViT model or a combination thereof.

[0108] Among them, the Bert model is a pre-training model.

[0109] Among them, the Chinese name of the BERT model is: Bidirectional Encoder Representations from Transformer, and the English name of the BERT model is: Bidirectional Encoder Representations from Transformer.

[0110] Among them, the Chinese name of the ViT model is: Vision Transformer, and the English name is: Vision Transformer.

[0111] S25, encoding the query sentence to generate a sentence vector;

[0112] Among them, sentence embedding not only retains the key semantic information in the query sentence, but also enables the query sentence to be compared and analyzed with other sentences in a computable way.

[0113] S26, based on a preset query method, in a preset vector database, using the sentence vector, query a processing solution for the abnormal information, and send the processing solution to the user equipment.

[0114] Among them, the vector database is a database system specifically used to store, query and process vector data.

[0115] The vector database includes a Milvus database, a Faiss database or a combination thereof.

[0116] Preferably, the vector database adopts the Milvus database.

[0117] Among them, the Milvus database is an open source vector database specifically used for storing, managing and searching large-scale vector data.

[0118] Among them, the Faiss database, the full name of Faiss is: Facebook AI Similarity Search.

[0119] Among them, the Faiss database is suitable for vector data of different scales and dimensions.

[0120] Among them, in the preset query mode, in a preset vector database, the processing scheme of the abnormal information is queried through the sentence vector, and after the processing scheme is sent to the user equipment, the abnormal information processing method includes:

[0121] A preset storage area is obtained, and the abnormal information and the processing solution are stored in the storage area.

[0122] In the embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, based on a preset query method, in the vector database, the processing solution of the exception information is queried through the sentence vector, and the processing solution is sent to the user device. Since no manual processing is required, the processing time of the exception information is reduced, which is conducive to improving the processing efficiency of the exception information. On the other hand, since the vector database can process query requests in parallel, the query time of the processing solution of the exception information is reduced, which is conducive to further improving the processing efficiency of the exception information.

[0123] See also Figure 3 , Figure 3 yes Figure 1 A specific implementation flow diagram of step S23 is described in detail as follows:

[0124] S31, processing the first feature vector and the second feature vector through a preset attention mechanism;

[0125] Among them, the attention mechanism is an important technology in deep learning, which simulates the ability of humans to automatically focus on important parts when processing information. The attention mechanism performs weighted processing on the input data, allowing the multimodal large model to dynamically adjust the attention to different parts, thereby extracting key information more effectively.

[0126] S32, concatenating the first feature vector and the second feature vector processed by the attention mechanism to generate a third feature vector.

[0127] In an embodiment of the present invention, since the attention mechanism can make the multimodal large model pay more attention to those features that have substantial contributions to the task, the first feature vector and the second feature vector processed by the attention mechanism are spliced ​​to generate a third feature vector, which can ensure that the spliced ​​third feature vector more concentratedly reflects the core content of the input data.

[0128] See also Figure 4 , Figure 4 yes Figure 1 A specific implementation flow diagram of step S25 is described in detail as follows:

[0129] S41, encoding each word of the query sentence through a preset word embedding model to generate a word vector corresponding to each word;

[0130] S42, combining the word vectors corresponding to each of the words to generate the sentence vector.

[0131] Exemplarily, combining the word vectors corresponding to each of the words to generate the sentence vector includes:

[0132] Using the Transformer encoder, the word vectors corresponding to each of the words are combined to generate the sentence vector.

[0133] Among them, Transformer is a neural network model based on the attention mechanism. Transformer can capture the long-distance dependencies between data when processing sequence data.

[0134] In the embodiment of the present invention, the sentence vector, as a numerical representation of the query statement, can capture the key information and semantic features in the query statement, so that the vector database processes the sentence vector.

[0135] See also Figure 5 , Figure 5 yes Figure 1 A specific implementation flow diagram of step S26 is described in detail as follows:

[0136] S51, packaging the sentence vector and the query instruction into a query request, and sending the query request to a preset vector database;

[0137] S52, receiving the query result returned by the vector database after executing the query request, selecting the query result as a processing solution for the abnormal information, and sending the processing solution to the user device, wherein the processing solution includes one of a text solution, an image solution, and a video solution, or a combination thereof.

[0138] Among them, the text scheme is a scheme in text format, and the image scheme is a scheme in image format.

[0139] The video solution is a solution in video format.

[0140] Among them, the vector database calculates the similarity between the sentence vector and the vector data stored in the vector database through a similarity calculation method. In the query result, the solutions corresponding to the vector data are arranged in order from high to low similarity to ensure that the solution ranked first in the query result is the solution corresponding to the vector data with the greatest similarity. The similarity calculation method includes Euclidean distance and cosine similarity.

[0141] Exemplarily, the receiving a query result returned after the vector database executes the query request, selecting the query result as a processing solution for the abnormal information, and sending the processing solution to the user equipment includes:

[0142] Receiving a query result returned by the vector database after executing the query request;

[0143] The solution ranked first in the query result is obtained as the solution for handling the abnormal information.

[0144] In the embodiment of the present invention, since the vector database can process query requests in parallel, the query time of the abnormal information processing solution is reduced, which is conducive to further improving the abnormal information processing efficiency.

[0145] See also Figure 6 , Figure 6 is a schematic diagram of a structure of an abnormal information processing device in one embodiment of the present invention, Figure 6 As shown, the abnormal information processing device includes an acquisition module 101, an extraction module 102, a first generation module 103, a determination module 104, a second generation module 105, and a processing module 106. The functional modules are described in detail as follows:

[0146] The acquisition module 101 is used to acquire the abnormal information sent by the user equipment, wherein the abnormal information includes the screen image and description text when the application of the user equipment is abnormal;

[0147] An extraction module 102, configured to perform feature extraction on the screen image to obtain a first feature vector, and perform feature extraction on the caption text to obtain a second feature vector;

[0148] A first generating module 103, configured to fuse the first feature vector and the second feature vector based on a preset fusion method to generate a third feature vector;

[0149] A determination module 104, configured to determine a query statement based on a preset multimodal macro model and the third feature vector;

[0150] A second generating module 105 is used to encode the query sentence and generate a sentence vector;

[0151] The processing module 106 is used to query the processing solution of the abnormal information in a preset vector database through the sentence vector based on a preset query method, and send the processing solution to the user equipment.

[0152] In one embodiment, the extraction module 102 includes:

[0153] A first extraction subunit, configured to use a preset image feature extraction network to perform feature extraction on the screen image to obtain a first feature vector;

[0154] The second extraction subunit is used to use a preset text feature extraction network to perform feature extraction on the caption text to obtain a second feature vector.

[0155] In one embodiment, the first generating module 103 includes:

[0156] a processing subunit, configured to process the first feature vector and the second feature vector through a preset attention mechanism;

[0157] A concatenation subunit is used to concatenate the first feature vector and the second feature vector processed by the attention mechanism to generate a third feature vector.

[0158] In one embodiment, the determining module 104 includes:

[0159] An input subunit, used for inputting the third feature vector into a preset multimodal macro model;

[0160] An acquisition subunit is used to acquire a query statement generated by the multimodal large model based on the third feature vector.

[0161] In one embodiment, the second generating module 105 includes:

[0162] A first generating subunit is used to encode each word of the query sentence through a preset word embedding model to generate a word vector corresponding to each word;

[0163] The second generating subunit is used to combine the word vectors corresponding to each of the words to generate the sentence vector.

[0164] In one embodiment, the processing module 106 includes:

[0165] A first sending subunit, used to package the sentence vector and the query instruction into a query request, and send the query request to a preset vector database;

[0166] The second sending sub-unit is used to receive the query result returned by the vector database after executing the query request, select the query result as the processing solution for the exception information, and send the processing solution to the user device, wherein the processing solution includes one of a text solution, an image solution, and a video solution, or a combination thereof.

[0167] In one embodiment, the abnormal information processing device further includes:

[0168] The storage module is used to obtain a preset storage area and store the abnormal information and the processing solution in the storage area.

[0169] In the embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, based on a preset query method, in the vector database, the processing solution of the exception information is queried through the sentence vector, and the processing solution is sent to the user device. Since no manual processing is required, the processing time of the exception information is reduced, which is conducive to improving the processing efficiency of the exception information. On the other hand, since the vector database can process query requests in parallel, the query time of the processing solution of the exception information is reduced, which is conducive to further improving the processing efficiency of the exception information.

[0170] For the specific limitations on the abnormal information processing device, please refer to the limitations on the abnormal information processing method above, which will not be repeated here.

[0171] Each module in the above-mentioned abnormal information processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0172] See also Figure 7 , Figure 7 is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided, and its internal structure diagram can be as follows Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus.

[0173] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user device through a network connection. When the computer program is executed by the processor, a function or step of an abnormal information processing method can be implemented.

[0174] See also Figure 8 , Figure 8 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided whose internal structure diagram can be as follows Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external task database via a network connection. When the computer program is executed by the processor, it can implement the functions or steps of an abnormal information processing method.

[0175] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor.

[0176] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions of the server and the user device in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0177] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0178] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0179] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0180] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0181] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for processing abnormal information, characterized in that: include: Acquire abnormal information sent by the user device, wherein the abnormal information includes a screen image and description text when an application of the user device has an abnormality; Performing feature extraction on the screen image to obtain a first feature vector, and performing feature extraction on the caption text to obtain a second feature vector; Based on a preset fusion method, the first feature vector and the second feature vector are fused to generate a third feature vector; Determine a query statement based on a preset multimodal macro model and the third feature vector; Encode the query sentence to generate a sentence vector; Based on a preset query method, in a preset vector database, a processing solution for the abnormal information is queried through the sentence vector, and the processing solution is sent to the user equipment.

2. The abnormal information processing method according to claim 1, characterized in that: The step of extracting features from the screen image to obtain a first feature vector and extracting features from the description text to obtain a second feature vector comprises: Using a preset image feature extraction network to extract features from the screen image to obtain a first feature vector; A preset text feature extraction network is used to extract features from the description text to obtain a second feature vector.

3. The abnormal information processing method according to claim 1, characterized in that: The step of fusing the first feature vector and the second feature vector based on a preset fusion method to generate a third feature vector includes: Processing the first feature vector and the second feature vector through a preset attention mechanism; The first feature vector and the second feature vector processed by the attention mechanism are concatenated to generate a third feature vector.

4. The abnormal information processing method according to claim 1, characterized in that: The determining of the query statement based on the preset multimodal macro model and the third feature vector includes: Inputting the third feature vector into a preset multimodal macro model; Obtain a query statement generated by the multimodal large model based on the third feature vector.

5. The abnormal information processing method according to claim 1, characterized in that: The step of encoding the query sentence to generate a sentence vector includes: Encode each word of the query sentence through a preset word embedding model to generate a word vector corresponding to each word; The word vectors corresponding to each of the words are combined to generate the sentence vector.

6. The abnormal information processing method according to claim 1, characterized in that: The querying of the processing scheme for the abnormal information in a preset vector database by using the sentence vector based on a preset query method, and sending the processing scheme to the user equipment includes: Packing the sentence vector and the query instruction into a query request, and sending the query request to a preset vector database; Receive the query result returned by the vector database after executing the query request, select the query result as the processing solution for the abnormal information, and send the processing solution to the user device, wherein the processing solution includes one of a text solution, an image solution, and a video solution, or a combination thereof.

7. The abnormal information processing method according to any one of claims 1 to 6, characterized in that: In the preset query mode, in a preset vector database, the processing scheme of the abnormal information is queried through the sentence vector, and after the processing scheme is sent to the user equipment, the abnormal information processing method includes: A preset storage area is obtained, and the abnormal information and the processing solution are stored in the storage area.

8. An abnormal information processing device, characterized in that: include: An acquisition module, used to acquire abnormal information sent by a user device, wherein the abnormal information includes a screen image and a description text when an abnormality occurs in an application of the user device; An extraction module, used for performing feature extraction on the screen image to obtain a first feature vector, and performing feature extraction on the caption text to obtain a second feature vector; A first generating module, configured to fuse the first feature vector and the second feature vector based on a preset fusion method to generate a third feature vector; A determination module, configured to determine a query statement based on a preset multimodal macro model and the third feature vector; A second generation module is used to encode the query sentence and generate a sentence vector; The processing module is used to query the processing solution of the abnormal information through the sentence vector in a preset vector database based on a preset query method, and send the processing solution to the user equipment.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the abnormal information processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the abnormal information processing method according to any one of claims 1 to 7 are implemented.

Images