Work order generation method, work order processing method and work order management system

Abstract

The invention discloses a work order generation method, a work order processing method and a work order management system. The work order generation method comprises the following steps: performing text extraction on multi-modal problem data reported by a user to obtain corresponding target text data; analyzing the multi-modal problem data by using the problem classification model to obtain a corresponding problem type, inputting the target text data into a work order template matched with the problem type to obtain a work order to be processed, and determining a generation timestamp of the work order to be processed; analyzing the multi-modal problem data by using an emotional tendency recognition model to obtain a corresponding emotional tendency; and packaging the emotional tendency, the question type, the work order to be processed and the corresponding generation timestamp corresponding to the multi-modal question data into a work order information packet, sending the work order information packet to a server, and receiving a question result fed back by the server. The technical problem that the work order generation accuracy is poor due to the fact that the multi-modal feedback data of the user cannot be fully understood in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a work order generation method, a work order processing method, and a work order management system. Background Art

[0002] In the telecommunications industry, handling customer tickets is crucial for maintaining customer satisfaction and improving service levels. Traditional ticket processing systems rely primarily on a single-modal input, typically text descriptions. This limits the depth and breadth of the system's understanding of the problem context. For example, when a customer describes a device malfunction solely through text, customer service personnel may struggle to accurately grasp the complexity of the issue, especially for situations that require visual or auditory information to assist understanding.

[0003] Recent advances in AI, particularly in natural language processing and computer vision, have made multimodal data processing possible. However, existing technologies for processing multimodal work orders are unable to accurately extract key information from complex data containing video, voice, images, and text to generate a work order describing the problem. Currently, no effective solution has been proposed to this problem. Summary of the Invention

[0004] The embodiments of the present application provide a work order generation method, a work order processing method, and a work order management system to at least solve the technical problem that the relevant technology cannot fully understand the user's multimodal feedback data, resulting in poor accuracy in generating work orders.

[0005] According to one aspect of an embodiment of the present application, a work order generation method is provided, which is applied to a client and includes: performing text extraction on multimodal problem data reported by a user to obtain corresponding target text data; using a problem classification model to analyze the multimodal problem data to obtain a corresponding problem type, and inputting the target text data into a work order template that matches the problem type to obtain a work order to be processed, and determining a generation timestamp of the work order to be processed; using a sentiment tendency recognition model to analyze the multimodal problem data to obtain a corresponding sentiment tendency; packaging the sentiment tendency, problem type, work order to be processed and corresponding generation timestamp corresponding to the multimodal problem data into a work order information package and sending it to a server, and receiving the problem results fed back by the server.

[0006] Optionally, text extraction is performed on the multimodal problem data reported by the user to obtain corresponding target text data, including: extracting problem data corresponding to different modal types from the multimodal problem data reported by the user, wherein the modal types include: text type, video type, voice type, and image type; for video problem data of video type, feature extraction is performed on the video problem data at multiple granularities according to frame segments to obtain the first frame vector of each frame segment at each granularity, and feature extraction is performed on the video problem data according to frame segments to obtain the second frame vector containing spatial position coding for each frame segment; summary extraction is performed on the video problem data according to frame segments to obtain the initial video summary of each frame segment; the second frame vector of each frame segment is input into a multi-layer encoder, and the encoding vector obtained by encoding the first frame vector of each frame segment at each granularity and the initial video summary of each frame segment is input into a multi-layer decoder, and then the decoding output result of each layer of decoder and the encoding output result of each layer of encoder are used as the input of the next decoder to obtain the decoding output after multiple decoding. Output result; Use video summary generation model to analyze the decoding output result to obtain multi-granularity video summary of video question data; For speech question data of speech type, perform first preprocessing on the speech question data, and use automatic speech recognition model to analyze the preprocessed speech question data to obtain corresponding first text data, wherein the first preprocessing at least includes: noise reduction and speech segmentation; For image question data of image type, perform second preprocessing on the image question data, and use optical character recognition model to analyze the preprocessed image question data to obtain corresponding second text data, wherein the second preprocessing at least includes: size adjustment, format conversion, and image enhancement; Perform third preprocessing on text question data of text type, multi-granularity video summary, first text data and second text data, and fuse the preprocessed text question data of text type, multi-granularity video summary, first text data and second text data to obtain corresponding target text data, wherein the third preprocessing at least includes: data cleaning, case format conversion, and data deduplication.

[0007] Optionally, summary extraction is performed on the video problem data according to frame segments to obtain an initial video summary of each frame segment, including: discretizing the video problem data into multiple frame segments, wherein each frame segment includes multiple continuous image frames; for each frame segment, using an optical flow algorithm to determine the motion activity index of each image frame in the frame segment, and taking the image frame with a motion activity index higher than a preset threshold value as a motion peak frame; discretizing the frame segment into at least one event atom based on the motion peak frame, wherein each event atom represents a coherent action sequence; for each event atom, extracting the visual features of the continuous image frames corresponding to the event atom, and converting the visual features into corresponding semantic descriptions to obtain a summary text of the event atom; splicing the summary text of each event atom in sequence according to the occurrence order of each event atom to obtain the initial video summary of the frame segment.

[0008] Optionally, the video summary generation model includes: a long short-term memory network, a feedforward neural network, and an output network, wherein the video summary generation model is used to analyze the decoding output result to obtain a multi-granularity video summary of the video problem data, including: using the long short-term memory network in the video summary generation model to analyze the decoding output result to obtain multiple candidate multi-granularity video summaries; using the feedforward neural network in the video summary generation model to evaluate each candidate multi-granularity video summary to obtain the confidence of each candidate multi-granularity video summary, wherein the confidence is comprehensively determined based on the characteristics of the candidate multi-granularity video summary and the correlation with the video problem data; using the output network in the video summary generation model to output the candidate multi-granularity video summary with the highest confidence to obtain a multi-granularity video summary of the video problem data.

[0009] Optionally, the multimodal question data is analyzed using a question classification model to obtain corresponding question types, including: when the multimodal question data only includes multiple other modal question data other than speech type voice question data, multiple feature extraction networks in the question classification model are used to perform feature extraction on the multiple other modal question data respectively to obtain feature maps of each other modality; the feature fusion unit in the question classification model is used to fuse the feature maps of each other modality to obtain a corresponding fused feature map; the deep feature extraction unit and the adaptive context convolution unit in the question classification model are used in turn to extract features from the fused feature map to obtain a multimodal feature vector containing deep features and context information; the output unit in the question classification model is used to analyze the multimodal feature vector to obtain the probability distribution of the multimodal feature vector belonging to various question categories, and the question category with the highest probability is taken as the first question type corresponding to the multimodal question data.

[0010] Optionally, the multimodal question data is analyzed using a question classification model to obtain the corresponding question type, and the method further includes: when the multimodal question data includes both speech question data of the speech type and multiple other modal question data, extracting the Mel-frequency cepstral coefficient features, high-order cumulative features and Mel-frequency cepstral coefficient distribution features of the speech question data; inputting the Mel-frequency cepstral coefficient features, high-order cumulative features and Mel-frequency cepstral coefficient distribution features of the speech question data into three channels in the question classification model respectively to obtain the feature vector output by each channel, wherein the types of channels include: convolutional neural network, long short-term memory network, and bidirectional long short-term memory network; using the feature fusion unit in the question classification model to output each channel The feature vectors of the first and second question types are fused to obtain a speech fusion feature vector; the output unit in the question classification model is used to analyze the speech fusion feature vector to obtain the probability distribution of the speech fusion feature vector belonging to various question categories, and the question category with the highest probability is used as the second question type corresponding to the speech question data; it is determined whether the first question type and the second question type are the same; when the first question type and the second question type are the same, one of the first question type and the second question type is used as the question type corresponding to the multimodal question number; when the first question type and the second question type are different, the question type with the highest question priority is selected from the first question type and the second question type according to a preset question priority rule as the question type corresponding to the multimodal question number.

[0011] Optionally, the sentiment tendency recognition model includes: a single-modal feature extraction unit and a single-modal feature fusion unit corresponding to each of the multiple modal types, a feature splicing unit, and a sentiment classification unit, wherein each single-modal feature extraction unit is used to extract the single-modal feature vector of the problem data of the corresponding modal type, and to extract the joint feature vector between the single-modal feature vector corresponding to the corresponding modal type and the single-modal feature vectors corresponding to other modal types; each single-modal feature fusion unit is used to use the attention mechanism to determine the weight coefficients of the single-modal feature vector and multiple joint feature vectors output by the single-modal feature extraction unit of the corresponding modal type, and use the weight coefficients to weightedly fuse the single-modal feature vector and multiple joint feature vectors output by the single-modal feature extraction unit of the corresponding modal type to obtain the corresponding single-modal fusion feature vector; the feature splicing unit is used to splice the single-modal fusion feature vectors output by the single-modal feature fusion units corresponding to each modal type to obtain a multimodal fusion feature vector; the sentiment classification unit is used to determine the probability distribution of the multimodal fusion feature vector belonging to various sentiment categories, and output the sentiment category with the highest probability as the prediction result.

[0012] According to one aspect of an embodiment of the present application, a work order processing method is provided, which is applied to a server and includes: receiving multiple work order information packets sent by multiple clients within a preset time period, wherein each work order information packet includes the sentiment tendency, problem type, pending work order and corresponding generation timestamp corresponding to the multimodal problem data reported by the corresponding client; parsing each work order information packet, and comprehensively determining the processing priority of each pending work order based on the sentiment tendency, problem type and generation timestamp of the pending work order in the parsing result; calling a preset large language model to process each pending work order in order of processing priority from high to low, obtaining the problem result of each pending work order, and feeding back each problem result to the corresponding client in turn.

[0013] According to another aspect of the embodiment of the present application, a work order management system is also provided, including: multiple clients and a server, wherein each client is used to perform text extraction on the multimodal problem data reported by the user to obtain corresponding target text data; use the problem classification model to analyze the multimodal problem data to obtain the corresponding problem type, and input the target text data into a work order template that matches the problem type to obtain a to-be-processed work order, and determine the generation timestamp of the to-be-processed work order; use the sentiment tendency recognition model to analyze the multimodal problem data to obtain the corresponding sentiment tendency; package the sentiment tendency, problem type, to-be-processed work order and the corresponding generation timestamp corresponding to the multimodal problem data into A work order information package is sent to a server; the server is used to receive multiple work order information packages sent by multiple clients within a preset time period, wherein each work order information package includes the sentiment tendency, problem type, pending work order and corresponding generation timestamp corresponding to the multimodal problem data reported by the corresponding client; each work order information package is parsed, and the processing priority of each pending work order is comprehensively determined based on the sentiment tendency, problem type and generation timestamp of the pending work order in the parsing result; in order of processing priority from high to low, the preset large language model is called to process each pending work order in turn, obtain the problem result of each pending work order, and feed back each problem result to the corresponding client in turn.

[0014] According to another aspect of an embodiment of the present application, a computer program product is further provided, comprising: a computer program, wherein when the computer program is executed by a processor, the above-mentioned work order generating method or work order processing method is implemented.

[0015] According to another aspect of an embodiment of the present application, an electronic device is further provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned work order generation method or work order processing method through the computer program.

[0016] In an embodiment of the present application, the client processes and analyzes the multimodal problem data reported by the user to obtain the corresponding text data, problem type and emotional tendency, and then fills the text data into the work order template that matches the problem type to obtain a work order to be processed. Finally, the emotional tendency, problem type, work order to be processed and the corresponding generation timestamp corresponding to the multimodal problem data are packaged into a work order information package and sent to the server, thereby achieving a comprehensive analysis and accurate classification of the multimodal problem data, while capturing the user's emotional state, and providing key information for subsequent work order processing, thereby achieving the purpose of improving the quality and processing efficiency of work order generation, and optimizing the user experience, thereby solving the technical problem that the relevant technology cannot fully understand the user's multimodal feedback data, resulting in poor accuracy in generating work orders. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a flowchart of an optional work order generation method according to an embodiment of the present application;

[0019] Figure 2 This is a flowchart of an optional work order processing method according to an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the structure of an optional work order management system according to an embodiment of the present application;

[0021] Figure 4 This is a hardware structure block diagram of a computer terminal that is optionally used to implement a work order generation method or a work order processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0023] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0024] Example 1

[0025] According to an embodiment of the present application, a work order generation method applied at a client is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] Figure 1 This is a flow chart of a work order generation method provided in accordance with an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0027] Step S102: performing text extraction on the multimodal question data reported by the user to obtain corresponding target text data.

[0028] Step S104: Analyze the multimodal question data using the question classification model to obtain the corresponding question type, input the target text data into the work order template that matches the question type, obtain the work order to be processed, and determine the generation timestamp of the work order to be processed.

[0029] Step S106: Analyze the multimodal question data using the sentiment tendency recognition model to obtain the corresponding sentiment tendency.

[0030] In step S108, the multimodal question data, including the sentiment, question type, pending work order, and corresponding generation timestamp, is packaged into a work order information package and sent to the server. The server parses the multiple work order information packages received within a preset time period, determines the processing priority of each pending work order based on the parsing results, and then uses the preset large language model to process each pending work order in descending order of priority, obtaining a problem result for each pending work order and feeding it back to the corresponding user.

[0031] Based on the scheme defined by the above steps S102 to S108, it can be known that in an embodiment of the present application, the client processes and analyzes the multimodal problem data reported by the user to obtain the corresponding text data, problem type and emotional tendency, and then fills the text data into the work order template that matches the problem type to obtain a work order to be processed. Finally, the emotional tendency, problem type, work order to be processed and the corresponding generation timestamp corresponding to the multimodal problem data are packaged into a work order information package and sent to the server, thereby achieving a comprehensive analysis and accurate classification of the multimodal problem data, while capturing the user's emotional state, providing key information for subsequent work order processing, and achieving the purpose of improving the quality of work order generation and processing efficiency, and optimizing the user experience.

[0032] The following describes the various steps of the work order generation method in conjunction with a specific implementation process.

[0033] In the technical solution provided in step S102 above, the client can obtain multimodal problem data, such as at least two of video problem data, voice problem data, image problem data, and text problem data, reported by the user through multiple methods (such as but not limited to text input, voice recording, image upload, and video recording) through one or more human-computer interaction interfaces. The problem data can be the user's desired feedback on the problem, such as a video uploaded by the user describing a mobile phone that cannot be turned off after long pressing the power button for a period of time, and a text question.

[0034] As an optional implementation, in the technical solution provided in step S102 above, the client may obtain the target text data corresponding to the multimodal question data according to the following method, including:

[0035] The client first extracts question data corresponding to different modal types from the multimodal question data reported by the user. Modal types include: text type, video type, voice type, and image type.

[0036] For video question data of the video type, the client can obtain the corresponding text data through the following methods:

[0037] Step 1: Perform feature extraction on the video problem data at multiple granularities, based on frame segments. This yields a first frame vector for each frame segment at each granularity. Feature extraction is also performed on the video problem data based on frame segments, yielding a second frame vector containing spatial position encoding for each frame segment. Granularity refers to the different levels of visual objects in the video content, based on spatial scale and semantic hierarchy. Therefore, granularity can be understood as the abstractness or level of detail of a feature.

[0038] Specifically, the client can first divide the video problem data into a series of frame segments based on time intervals; within each frame segment, the client can extract feature vectors at different granularities. This multi-granularity feature extraction method can simultaneously capture information from both subtle differences in the video and a macro perspective, ensuring that even video content in rapidly changing or complex scenes can be accurately interpreted, thereby more accurately identifying problems or customer needs in the video; at the same time, the client can use the ViT (Vision Transformer) model to extract the feature vectors of each frame segment in the video problem data (the vector encodes the key visual information of continuous image frames in the frame segment), and add the spatial position code of the frame segment to the feature vector to form a second frame vector carrying the position code.

[0039] Step 2: Extract summary of the video problem data according to frame segments to obtain the initial video summary of each frame segment to capture the key events or time points in the frame segment.

[0040] Regarding the extraction of the initial video summary, the client may implement it according to the following method: discretize the video question data into a plurality of frame segments, wherein each frame segment includes a plurality of continuous image frames. For each frame segment, an optical flow algorithm is used to determine the motion activity index of each image frame in the frame segment, and image frames with motion activity indexes higher than a preset threshold are regarded as motion peak frames. The motion activity index reflects the degree of motion change between image frames. Once the motion activity index of an image frame exceeds the preset threshold, the corresponding image frame is marked as a motion peak frame. The motion peak frame usually corresponds to the moment when the action change in the video is most drastic or significant. Based on the motion peak frame, the frame segment is discretized into at least one event atom, where each event atom represents a coherent action sequence (such as "the user presses and holds the power button", "the indicator light turns red", etc.), starting from a change in an action and ending with the next significant change (the appearance of a new motion peak frame). For each event atom, a deep learning model is used to extract the visual features of the consecutive image frames corresponding to the event atom, and a speech understanding model (such as Transformer) is used to convert the visual features into corresponding semantic descriptions to obtain a summary text of the event atom. The summary texts of each event atom are spliced ​​in the order of occurrence of each event atom to obtain the initial video summary of the frame segment.

[0041] Among them, the optical flow algorithm analyzes the pixel changes between two consecutive image frames in the frame segment, estimates the position of each pixel in the next frame, and thus calculates the optical flow vector of the pixel; then calculates the average value of the optical flow vector amplitude (that is, the size of the optical flow vector) of all pixels in each image frame, or calculates the ratio of all non-zero optical flow vectors in each image frame, to obtain the motion activity index of each image frame.

[0042] Step 3: Input the second frame vector of each frame segment into the multi-layer encoder, and input the encoding vector obtained by encoding the first frame vector of each frame segment at each granularity and the initial video summary of each frame segment into the multi-layer decoder. Then, the decoding output result of each layer of decoder and the encoding output result of each layer of encoder are used as the input of the next decoder. This process is carried out layer by layer, and finally the decoding output result after multiple decoding is obtained, which contains a multi-dimensional interpretation of the video content.

[0043] Step 4: Use the video summary generation model to analyze the decoding output results to obtain a multi-granular video summary of the video question data. The video summary generation model includes: a long short-term memory network, a feedforward neural network, and an output network.

[0044] Specifically, the video summary generation model analyzes the decoding output results as follows: the client can first use the long short-term memory network in the video summary generation model to analyze the decoding output results to obtain multiple candidate multi-granularity video summaries. The long short-term memory network can capture the long-term dependencies of sequence data and perform sequence modeling on the decoding output results to generate multiple possible candidate video summaries. These candidate video summaries reflect the different description angles and complexity of the video events and contain key information points extracted from the video frame fragments. Then, the feedforward neural network in the video summary generation model is used to evaluate each candidate multi-granularity video summary to obtain the confidence of each candidate multi-granularity video summary. The confidence is determined based on the characteristics of the candidate multi-granularity video summary (i.e., its own coherence and information content) and its relevance to the video question data. Therefore, the better the coherence of the candidate multi-granularity video summary, the richer the information content, and the higher the relevance to the question, the higher the corresponding confidence; otherwise, the lower the confidence. The output network in the video summary generation model outputs the candidate multi-granularity video summary with the highest confidence, thereby obtaining the multi-granularity video summary of the video question data.

[0045] For voice question data of the voice type, the client can obtain the corresponding text data through the following methods:

[0046] The voice question data is first preprocessed, and the preprocessed voice question data is analyzed using an automatic speech recognition model to obtain corresponding first text data.

[0047] Specifically, the client preprocesses the voice data, including noise reduction and speech segmentation. Noise reduction utilizes noise suppression algorithms to reduce background noise and interference, improving speech clarity. Speech segmentation divides continuous voice data into separate segments, each containing a complete sentence or expression. Furthermore, the automatic speech recognition model, based on a deep learning architecture, is trained using a large amount of speech-to-text training data to identify the speech characteristics of different speakers, thereby converting the speech data into text.

[0048] For image question data of image type, the client can obtain the corresponding text data through the following methods:

[0049] The image question data is subjected to a second preprocessing, and the preprocessed image question data is analyzed using an optical character recognition model to obtain corresponding second text data.

[0050] Specifically, the client first preprocesses the image problem data, including resizing, format conversion, and image enhancement. Resizing refers to adjusting the image problem data to a uniform size to accommodate the model's input requirements; format conversion refers to converting the image problem data to a format readable by the model, such as JPEG or PNG; and image enhancement refers to utilizing image enhancement techniques such as contrast adjustment, sharpening, and denoising to enhance the clarity of text in the image and reduce recognition errors. Furthermore, the optical character recognition model, based on a deep learning architecture, is trained using a large amount of image-text training data to recognize different fonts and text layout features, thereby converting image data into text data.

[0051] Finally, the text question data of the text type, the multi-granularity video summary, the first text data and the second text data are subjected to a third preprocessing, and the preprocessed text question data of the text type, the multi-granularity video summary, the first text data and the second text data are fused to obtain the corresponding target text data.

[0052] The third preprocessing mentioned above includes at least: data cleaning, case conversion, data deduplication, etc., wherein data cleaning includes removing unnecessary symbols, characters and spaces, and applying stem extraction or synonym extraction to standardize the expression; case conversion is to convert all texts into a unified case format; data deduplication is to remove repeated text content or keywords from all texts to ensure that the subsequent analysis is only considered once, avoid deviations in the analysis process, and reduce the amount of subsequent processing data. All preprocessed texts are integrated together to form a larger text set, thereby obtaining a target text data containing multimodal information, which can accurately reflect all the details of the problem reported by the user.

[0053] It should be noted that in order to efficiently integrate text-type text question data, multi-granularity video summaries, first text data and second text data, the word frequency-inverse document frequency algorithm can be used to extract keywords from these text data, and these keywords can be integrated to obtain a target text data containing multimodal information.

[0054] Furthermore, the client can use the question classification model to analyze the multimodal question data to obtain the corresponding question type. According to the modality type contained in the multimodal question data, it can be divided into the following two implementation methods:

[0055] 1. When the multimodal question data includes only voice question data in addition to the voice type, multiple other modal question data (such as at least two of video question data, image question data, and text question data) can be obtained by the client using the following method to obtain the question type corresponding to the multimodal question data:

[0056] Step 1: Use multiple feature extraction networks in the question classification model to extract features from multiple other modal question data to obtain the feature maps of each other modality.

[0057] Each feature extraction network is used to extract feature maps of problem data of a certain modality, and each feature extraction network includes at least: multiple convolutional layers for extracting local features, multiple pooling layers for reducing feature dimensions, and a fully connected layer for combining local features. For example, the feature extraction network may include thirteen convolutional layers, thirteen ReLU activation layers, four pooling layers, and one fully connected layer, and a ReLU activation layer is added after each convolutional layer, the first pooling layer is added after the second ReLU activation layer, the second pooling layer is added after the fourth ReLU activation layer, the third pooling layer is added after the seventh ReLU activation layer, the fourth pooling layer is added after the tenth ReLU activation layer, and a fully connected layer is added after the last ReLU activation layer. The fully connected layer uses a 1×1 convolution kernel to process the feature map, and the 1×1 convolution kernel can change the number of channels of the feature map without changing its spatial dimension, and integrate the local features extracted by different convolutional layers to achieve global feature combination and high-level feature extraction, thereby obtaining a feature map of the corresponding modality.

[0058] Step 2: Use the feature fusion unit in the question classification model to fuse the feature maps of each other modality to obtain the corresponding fused feature map.

[0059] Among them, the feature fusion unit can use the attention mechanism to determine the weights of the feature maps of each other modality to perform weighted summation to obtain a fused feature map.

[0060] Step 3: Use the deep feature extraction unit and adaptive context convolution unit in the question classification model to extract features from the fused feature map in turn to obtain a multimodal feature vector containing deep features and context information.

[0061] Specifically, the above-mentioned deep feature extraction unit can be a dynamic multi-scale convolutional network (Dynamic Multi-Scale Convolutional Network, DMSCNet), which includes multiple convolutional layers, residual blocks and dynamic multi-scale convolution sub-units connected in sequence.

[0062] (1) Multiple convolutional layers including 3×3 convolutional layers and 5×5 convolutional layers, with a ReLU activation layer added after each convolutional layer;

[0063] (2) In the residual block, one or more standard convolutional layers are replaced by dynamic multi-scale convolutional layers. Therefore, the expression of the residual block can be written as:

[0064] F residual =DMSC(ReLU(Conv(F input )))

[0065] Where F input Represents the feature map input to the current residual block, F residual Represents the feature map output by the residual block, which contains multi-scale feature information. Conv represents the convolution operation, and DMSC represents the dynamic multi-scale convolution operation.

[0066] (3) The expression of the dynamic multi-scale convolution operation performed in the dynamic multi-scale convolution subunit can be written as:

[0067]

[0068] Where F′ input represents the input vector of the dynamic multi-scale convolutional layer, F DMSC Represents the feature vector output by the dynamic multi-scale convolutional layer, w k Represents the original weight of the kth dynamic convolution kernel, Conv(F′ input ,w k ) represents the convolution operation using the kth dynamic convolution kernel, k represents the kth dynamic convolution kernel, K represents the number of dynamic convolution kernels, α k It represents the weight generated by the attention mechanism in the dynamic multi-scale convolutional layer, and its expression can be written as:

[0069]

[0070] Where β k =f(F input; θ) represents the attention weight, f represents a fully connected layer network for generating attention weight (its input is the feature map and the output is the weight parameter β k , these weight parameters represent the weights β of the input feature map on different convolution kernels k ), θ represents the parameter of f, and the Softmax function is used to normalize the output of the fully connected layer network to α k (make sure the sum of these weight parameters is 1), that is, α k It is calculated by calculating the attention weight β of the kth dynamic convolution kernel k The exponential value of all the first k attention weights β j The ratio of the sum of the exponential values ​​is determined, so α k The current attention weight β can be normalized k The exponential value of α k Represents the relative weight of the kth dynamic convolution kernel among all dynamic convolution kernels.

[0071] In addition, the above-mentioned adaptive context convolution unit integrates global context features and local context features by introducing a context-aware mechanism to improve the model's perception of scenes and objects. Therefore, the expression of the adaptive context convolution unit can be written as:

[0072] y=Concat(GlobalContext(x),LocalContext(x))

[0073] Where GlobalContext(x) represents the global context feature, LocalContext(x) represents the local context feature, and Concat represents the concatenation operation.

[0074] Step 4: Use the output unit in the question classification model to analyze the multimodal feature vector to obtain the probability distribution of the multimodal feature vector belonging to various question categories, and use the question category with the highest probability as the first question type corresponding to the multimodal question data.

[0075] The output unit includes a classification subunit and a regression subunit. The classification subunit includes a fully connected layer and a Softmax layer connected in sequence, while the regression subunit includes a fully connected layer and a linear layer connected in sequence. Therefore, in the classification subunit, the fully connected layer is used to process the multimodal feature vector to compress the high-dimensional feature vector into a lower-dimensional space. The low-dimensional feature vector processed by the fully connected layer is then mapped into a probability distribution containing the probability values ​​of each question category through the Softmax function in the linear layer. The sum of all probabilities in this probability distribution is 1. The question category with the highest probability in the probability distribution is taken as the first question type corresponding to the multimodal question data.

[0076] 2. When the multimodal question data includes both voice question data of the voice type and multiple question data of other modalities, the client can obtain the question type corresponding to the multimodal question data in the following way:

[0077] Step 1: Use the question classification model to analyze the voice question data of the voice type and obtain the second question type corresponding to the voice question data. The specific implementation method is as follows:

[0078] First, extract the Mel-frequency cepstral coefficient features, high-order cumulative features, and Mel-frequency cepstral coefficient distribution features of the speech problem data.

[0079] The Mel-frequency cepstral coefficient feature can be obtained according to the following method: first pre-process the speech problem data, and divide the processed speech signal into short-time frames (each frame is usually 25-40ms, and the frame shift is 10-16ms), wherein the pre-processing at least includes removing DC classification, pre-emphasis (to enhance the high-frequency part), etc.; perform fast Fourier transform on each short-time frame to convert the time domain signal into a frequency domain signal to obtain the amplitude spectrum corresponding to each short-time frame, and use a group of triangular filter groups uniformly distributed on the Mel scale to filter the amplitude spectrum corresponding to each short-time frame to obtain the energy spectrum corresponding to each short-time frame output by the filter group; perform log operation on the energy spectrum corresponding to each short-time frame, and perform discrete cosine transform to obtain Mel-frequency cepstral coefficients; finally, select multi-level Mel-frequency cepstral coefficients (such as the first 12-13 levels) as the Mel-frequency cepstral coefficient feature of the speech problem data.

[0080] The high-order cumulant features can be obtained as follows: first preprocess the speech problem data and divide the processed speech signal into short-time frames (each frame is usually 25-40ms, with a frame shift of 10-16ms), where the preprocessing includes at least pre-emphasis and centering (removing the DC component); for each short-time frame, calculate the high-order cumulant of the short-time frame. Generally, the third-order cumulant and the fourth-order cumulant can be calculated. This is because the third-order cumulant can reveal the skewness of the signal, while the fourth-order cumulant (or kurtosis) can reflect the peak and flatness of the signal.

[0081] The distribution characteristics of the Mel-frequency cepstral coefficients (i.e., the entropy value of the Mel-frequency cepstral coefficients) can be obtained as follows: for each level of Mel-frequency cepstral coefficients, the occurrence probability value of the Mel-frequency cepstral coefficient is calculated and regarded as a discrete random variable; the entropy value of the Mel-frequency cepstral coefficient is determined by combining the occurrence probability values ​​of the Mel-frequency cepstral coefficients at each level with the entropy concept in information theory.

[0082] Next, the Mel-frequency cepstral coefficient features, high-order cumulant features, and Mel-frequency cepstral coefficient distribution features of the speech question data are respectively input into the three channels of the question classification model to obtain the feature vector output by each channel. The channel types include: convolutional neural network, long short-term memory network, and bidirectional long short-term memory network.

[0083] Then, the feature fusion unit in the question classification model is used to fuse the feature vectors output by each channel (i.e., element-wise addition) to obtain a speech fusion feature vector.

[0084] Finally, the output unit in the question classification model is used to analyze the speech fusion feature vector to obtain the probability distribution of the speech fusion feature vector belonging to various question categories, and the question category with the highest probability is used as the second question type corresponding to the speech question data.

[0085] Among them, the output unit also includes a fully connected layer and a linear layer. The fully connected layer is first used to process the speech fusion feature vector to compress the high-dimensional speech fusion feature vector into a lower-dimensional space, and then the low-dimensional feature vector processed by the fully connected layer is mapped into a probability distribution containing the probability values ​​of each question category through the Softmax function in the linear layer. The sum of all probabilities in the probability distribution is 1, and the question category with the highest probability in the probability distribution is taken as the second question type corresponding to the speech question data.

[0086] Step 2: Determine whether the first question type and the second question type are the same, wherein, if they are the same, execute the following step 3, otherwise execute the following step 4.

[0087] Step 3: Select one of the first and second question types as the question type corresponding to the multimodal question data. This consistency decision is based on the fact that the nature of the questions is the same whether viewed from the non-speech mode or the speech mode. Therefore, it is sufficient to select any one type as the question type corresponding to the multimodal question data.

[0088] Step 4: According to the preset problem priority rules, the problem type with the highest problem priority is selected from the first problem type and the second problem type as the problem type corresponding to the multimodal problem number. This inconsistent decision is because the non-voice mode and the voice mode point to different problem types. At this time, the client can judge according to the problem priority rules, where the problem priority rules are based on historical data, business experience or specific business needs to formulate priorities for different problem types (fault type problems > operation type problems > pricing type problems), which are used to select the problem type with the highest priority among multiple possible problem types. For example, if the first problem type is a pricing problem and the second problem type is an operation type problem, then the operation type problem can eventually be used as the problem type corresponding to the multimodal problem number.

[0089] After determining the problem type corresponding to the number of multimodal problems using the above method, the client can determine the work order template that matches the problem type from the preset work order template library and input the target text data extracted from the multimodal problem data (i.e., key information extracted from video, voice, image, and text data) into the matching work order template. The work order template library includes pre-designed work order templates for different problem types, which contain specific fields and information structures to ensure the standardization and completeness of the work order content. In addition, when the client generates a pending work order, it automatically records the generation timestamp of the pending work order.

[0090] In the technical solution provided in step S106 above, the client can also use an emotional tendency recognition model to analyze the multimodal question data to obtain corresponding emotional tendencies, where emotional tendencies include but are not limited to: anger, anxiety, impatience, peace, happiness, etc. The client can then judge the user's urgency about the question based on the emotional tendencies.

[0091] Specifically, the above-mentioned emotion tendency recognition model includes: a single-modal feature extraction unit and a single-modal feature fusion unit, a feature splicing unit, and an emotion classification unit corresponding to each of the multiple modal types, wherein:

[0092] Each unimodal feature extraction unit is configured to extract a unimodal feature vector of the question data of the corresponding modality type, and to extract a joint feature vector between the unimodal feature vector corresponding to the corresponding modality type and the unimodal feature vectors corresponding to other modality types. For example, the speech modality feature extraction unit may extract a unimodal feature vector corresponding to the speech modality, and extract a joint visual-audio feature vector between it and the unimodal feature vector corresponding to the video modality, an image-audio joint feature vector between it and the unimodal feature vector corresponding to the image modality, and a text-audio joint feature vector between it and the unimodal feature vector corresponding to the text modality.

[0093] Among them, the unimodal feature extraction unit corresponding to the speech type can be a wav2vec-large model or other specific mathematical models; the unimodal feature extraction unit corresponding to the video type can be a ViT (Vision Transformer) model or other convolutional neural networks, etc.; the unimodal feature extraction unit corresponding to the image type can be a convolutional neural network such as AlexNet, VGG, ResNet, etc.; the unimodal feature extraction unit corresponding to the text type can be a DeBERTa-large model, a bag-of-words model, etc. The embodiment of this application does not impose any specific restrictions on the model selection.

[0094] Each unimodal feature fusion unit is used to determine the weight coefficients of the unimodal feature vector and multiple joint feature vectors output by the unimodal feature extraction unit of the corresponding modality type using the attention mechanism, and use the weight coefficients to perform weighted fusion on the unimodal feature vector and multiple joint feature vectors output by the unimodal feature extraction unit of the corresponding modality type to obtain the corresponding unimodal fusion feature vector.

[0095] Taking the unimodal feature fusion unit corresponding to the speech type as an example, the calculation formulas for its unimodal feature vector and the weight coefficients of multiple joint feature vectors are as follows:

[0096]

[0097] Where a represents the voice question data of the voice type, b represents the text question data of the text type, c represents the video question data of the video type, and d represents the image question data corresponding to the image type; X a represents the unimodal feature vector corresponding to the speech problem data, q represents the unimodal feature fusion unit of the speech type, MLP represents the multi-layer perceptron, P a represents the set of modal scores for speech question data, represents the score of the speech problem data, represents the score of the audio-text modality, represents the score of the audio-visual modality, represents the score of the audio-image modality; softmax represents the normalized exponential function. When i is a, Represents the weight coefficient of the unimodal feature vector of the speech problem data. When i is b, Represents the weight coefficient of the joint feature vector of audio and text. When i is c, Represents the weight coefficient of the joint feature vector of audio and video. When i is d, The weight coefficient representing the joint feature vector of audio and image.

[0098] It should be noted that the weight coefficients of the unimodal feature vectors and multiple joint feature vectors corresponding to different modality types can be different, and for scenarios where the modality is missing, the unimodal fusion feature vectors of the missing modal data can be padded with 0 to achieve the unification of the feature dimension of the spliced ​​unimodal fusion feature vectors.

[0099] The feature splicing unit is used to splice the single-modal fusion feature vectors output by the single-modal feature fusion units corresponding to each modality type to obtain a multi-modal fusion feature vector.

[0100] The emotion classification unit is used to determine the probability distribution of the multimodal fusion feature vector belonging to various emotion categories, and output the emotion category with the highest probability as the prediction result.

[0101] The emotion tendency recognition model of the above structure can be obtained by iteratively training a neural network containing single-modal feature extraction units and single-modal feature fusion units, feature splicing units, and emotion classification units corresponding to multiple modal types using preset training sample data, and the training sample data includes: multimodal data and corresponding real emotion labels.

[0102] The above-mentioned emotion tendency recognition model can dynamically fuse the feature vectors of all modalities and use the complete feature vectors to realize emotion recognition, overcoming the problem of poor accuracy of emotion tendency analysis results when there is only one modality data.

[0103] It should be noted that the embodiment of the present application does not impose any specific restrictions on the execution order of step S104 and step S106. Step S104 can be executed first and then step S106, or step S106 can be executed first and then step S104, or step S104 and step S106 can be executed in parallel.

[0104] Finally, the client can package the sentiment tendency, problem type, pending work order and corresponding generation timestamp corresponding to the multimodal problem data reported by the user into work order information, and then send it to the server for work order processing.

[0105] Therefore, through the above-mentioned work order generation method, the client can accurately extract the problem text, intelligently classify the problem type, automatically fill in the work order template, and efficiently assess the emotional tendency from the multimodal problem data reported by the user, thereby achieving the efficiency and accuracy of work order generation.

[0106] Example 2

[0107] According to an embodiment of the present application, a work order processing method applied at a server is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0108] Figure 2 This is a flowchart of a work order processing method provided in accordance with an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0109] Step S202: Receive multiple work order information packets sent by multiple clients within a preset time period, wherein each work order information packet includes the sentiment tendency, problem type, pending work order, and corresponding generation timestamp corresponding to the multimodal problem data reported by the corresponding client.

[0110] In step S204, each work order information package is parsed, and the processing priority of each work order to be processed is comprehensively determined based on the sentiment tendency, problem type and generation timestamp of the work order to be processed in the parsing result.

[0111] In step S206, the preset large language model is called to process each pending work order in order of processing priority from high to low, to obtain the problem result of each pending work order, and each problem result is fed back to the corresponding client in turn.

[0112] The following describes the various steps of the work order processing method in conjunction with the specific implementation process.

[0113] In the technical solution provided in the above step S202, the server continuously monitors the network connection and waits to receive the work order information packets sent by the client. Once the work order information packets are received from multiple clients within a preset time period, they will be temporarily stored in the receiving buffer to ensure the integrity and order of the data.

[0114] In the technical solution provided in step S204 above, the server first parses the content of each received ticket information packet to extract key information, including sentiment, issue type, pending ticket, and corresponding creation timestamp. Next, the server determines the processing priority of each pending ticket based on the sentiment, issue type, and creation timestamp of the pending ticket.

[0115] That is to say, the server performs a weighted summation of the sentiment tendency, problem type, and generation timestamp of the pending work order to obtain the priority score of each pending work order, and uses this to determine the processing priority of each pending work order. Among them, the sentiment tendency reflects the user's emotional state (which reflects the user's urgency in processing the multimodal problem data), the problem type identifies the importance of the pending work order, and the generation timestamp provides the timeliness information of the pending work order. Therefore, the weight coefficients of these three parameters are the weights of sentiment tendency, problem type importance, and time sensitivity, respectively.

[0116] In the technical solution provided in the above step S206, the server can call the preset large language model to process each pending work order in order from high to low processing priority, obtain the problem result of each pending work order, and feed back each problem result to the corresponding client in turn.

[0117] The above steps can be understood as the server constructing a work order queue in descending order of priority based on the comprehensively determined processing priority; for the first pending work order in the queue, the server can first standardize the pending work order to convert it into a format that can be processed by the model, such as a standardized problem description; then dispatch the preset large language model to analyze the processed pending work order to retrieve the answer to the question closest to the description of the pending work order in the question and answer knowledge base associated with the large language module (containing various types of questions and corresponding problem results), and feed the answer back to the corresponding client for user query, thereby greatly improving the work order processing efficiency and customer satisfaction. At the same time, the server will continuously update the processing status of each pending work order to facilitate subsequent tracking and analysis.

[0118] After completing the processing of the current pending work order, the server will automatically select the next highest priority pending work order from the work order queue and repeat the above steps of calling the large language model and problem result feedback until all work orders are processed.

[0119] It's important to note that while the server is processing tickets, if new high-priority tickets appear, the server can dynamically adjust the order of the ticket queue to ensure that urgent and important tickets are processed first. Furthermore, the server dynamically updates its Q&A knowledge base based on the processing results and user feedback of each ticket in the ticket queue, using a large language model to continuously optimize the accuracy and efficiency of problem handling.

[0120] Based on the scheme defined in the above steps S202 to S206, it can be known that in the embodiment of the present application, the server adopts a work order processing scheme driven by deep learning and assisted by a large language model, which achieves the technical effect of comprehensively considering the work order processing order from multiple dimensions, intelligently identifying and responding to user issues, and achieving the purpose of improving work order processing efficiency, optimizing customer experience and enhancing service accuracy.

[0121] Example 3

[0122] According to an embodiment of the present application, a work order management system is also provided. Figure 3 As shown, the system includes: multiple clients 32i (i=1, 2, ..., n) and a server 34, wherein:

[0123] Each client 32i is used to perform text extraction on the multimodal problem data reported by the user to obtain the corresponding target text data; use the problem classification model to analyze the multimodal problem data to obtain the corresponding problem type, and input the target text data into the work order template that matches the problem type to obtain the work order to be processed, and determine the generation timestamp of the work order to be processed; use the sentiment tendency recognition model to analyze the multimodal problem data to obtain the corresponding sentiment tendency; package the sentiment tendency, problem type, work order to be processed and the corresponding generation timestamp corresponding to the multimodal problem data into a work order information package and send it to the server 34.

[0124] The server 34 is used to receive multiple work order information packages sent by multiple clients within a preset time period, wherein each work order information package includes the sentiment tendency, problem type, pending work order and corresponding generation timestamp corresponding to the multimodal problem data reported by the corresponding client; parse each work order information package, and comprehensively determine the processing priority of each pending work order based on the sentiment tendency, problem type and generation timestamp of the pending work order in the parsing result; call the preset large language model to process each pending work order in order of processing priority from high to low, obtain the problem result of each pending work order, and feed back each problem result to the corresponding client 32i in turn.

[0125] It should be noted that the multiple clients in the work order management system in the embodiment of the present application correspond one-to-one to the various implementation steps of the work order generation method in Example 1, and the server in the work order management system in the embodiment of the present application corresponds one-to-one to the various implementation steps of the work order processing method in Example 2. Since the work order generation method and the work order processing method have been described in detail in Examples 1 and 2, some details not reflected in this embodiment can be referred to Examples 1 and 2, and will not be elaborated here.

[0126] Example 4

[0127] According to an embodiment of the present application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, the work order generation method in Example 1 and the work order processing method in Example 2 are implemented.

[0128] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the work order generation method in Example 1 and the work order processing method in Example 2 by running the computer program.

[0129] According to an embodiment of the present application, a processor is further provided, which is used to run a computer program, wherein the work order generation method in Example 1 and the work order processing method in Example 2 are executed when the computer program is running.

[0130] According to an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the work order generation method in Example 1 and the work order processing method in Example 2 through the computer program.

[0131] Specifically, when the computer program is running, the following steps are executed: text extraction is performed on the multimodal problem data reported by the user to obtain corresponding target text data; the multimodal problem data is analyzed using a problem classification model to obtain a corresponding problem type, and the target text data is input into a work order template that matches the problem type to obtain a work order to be processed, and a generation timestamp of the work order to be processed is determined; the multimodal problem data is analyzed using a sentiment tendency recognition model to obtain a corresponding sentiment tendency; the sentiment tendency, problem type, work order to be processed and the corresponding generation timestamp corresponding to the multimodal problem data are packaged into a work order information package and sent to the server, and the problem results fed back by the server are received.

[0132] Specifically, the computer program executes the following steps when it is running: receiving multiple work order information packets sent by multiple clients within a preset time period, wherein each work order information packet includes the sentiment tendency, problem type, pending work order and corresponding generation timestamp corresponding to the multimodal problem data reported by the corresponding client; parsing each work order information packet, and comprehensively determining the processing priority of each pending work order based on the sentiment tendency, problem type and generation timestamp of the pending work order in the parsing result; calling the preset large language model to process each pending work order in order of processing priority from high to low, obtaining the problem result of each pending work order, and feeding back each problem result to the corresponding client in turn.

[0133] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 4 The hardware structure block diagram of a computer terminal for implementing a work order generation method or a work order processing method is shown. Figure 4As shown, the computer terminal 40 may include one or more (402a, 402b, ..., 402n are shown in the figure) processors 402 (the processor 402 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 404 for storing data, and a transmission device 406 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal 40 may also include Figure 4 More or fewer components than shown, or with Figure 4 Different configurations shown.

[0134] It should be noted that the one or more processors 402 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 40. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0135] The memory 404 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the work order generation method in the embodiment of the present application. The processor 402 executes various functional applications and data processing by running the software programs and modules stored in the memory 404, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 404 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 404 may further include a memory remotely located relative to the processor 402, and these remote memories may be connected to the computer terminal 40 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0136] Transmission device 406 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of computer terminal 40. In one embodiment, transmission device 406 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 406 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0137] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 40 .

[0138] The serial numbers of the above embodiments are for description only and do not represent the advantages or disadvantages of the embodiments.

[0139] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0141] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0142] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0144] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A work order generation method, characterized in that: Applied to the client, including: Perform text extraction on the multimodal question data reported by users to obtain the corresponding target text data; Analyzing the multimodal question data using a question classification model to obtain a corresponding question type, inputting the target text data into a work order template matching the question type to obtain a work order to be processed, and determining a generation timestamp of the work order to be processed; Analyzing the multimodal question data using a sentiment tendency recognition model to obtain corresponding sentiment tendencies; The emotional tendency, question type, pending work order and corresponding generation timestamp corresponding to the multimodal question data are packaged into a work order information package and sent to the server, and the question result fed back by the server is received.

2. The method according to claim 1, characterized in that Perform text extraction on the multimodal problem data reported by users to obtain the corresponding target text data, including: Extracting question data corresponding to different modal types from the multimodal question data reported by the user, wherein the modal types include: text type, video type, voice type, and image type; For the video problem data of the video type, feature extraction is performed on the video problem data at multiple granularities according to frame segments to obtain a first frame vector of each frame segment at each granularity, and feature extraction is performed on the video problem data according to frame segments to obtain a second frame vector containing a spatial position code for each frame segment; summary extraction is performed on the video problem data according to frame segments to obtain an initial video summary of each frame segment; the second frame vector of each frame segment is input into a multi-layer encoder, and the encoding vector obtained by encoding the first frame vector of each frame segment at each granularity and the initial video summary of each frame segment is input into a multi-layer decoder, and the decoding output result of each layer of decoder and the encoding output result of each layer of encoder are used as input to the next decoder to obtain a decoding output result after multiple decoding; the decoding output result is analyzed using a video summary generation model to obtain a multi-granularity video summary of the video problem data; For the speech problem data of the speech type, performing a first preprocessing on the speech problem data, and analyzing the preprocessed speech problem data using an automatic speech recognition model to obtain corresponding first text data, wherein the first preprocessing includes at least: noise reduction and speech segmentation; For the image problem data of the image type, performing a second preprocessing on the image problem data, analyzing the preprocessed image problem data using an optical character recognition model to obtain corresponding second text data, wherein the second preprocessing at least includes: size adjustment, format conversion, and image enhancement; The text question data of the text type, the multi-granularity video summary, the first text data and the second text data are subjected to a third preprocessing, and the preprocessed text question data of the text type, the multi-granularity video summary, the first text data and the second text data are fused to obtain corresponding target text data, wherein the third preprocessing includes at least: data cleaning, case format conversion, and data deduplication.

3. The method according to claim 2, characterized in that Extracting a summary of the video problem data according to the frame segments to obtain an initial video summary of each frame segment includes: Discretizing the video problem data into a plurality of frame segments, wherein each of the frame segments includes a plurality of continuous image frames; For each of the frame segments, an optical flow algorithm is used to determine the motion activity index of each of the image frames in the frame segment, and the image frame whose motion activity index is higher than a preset threshold value is used as the motion peak frame; based on the motion peak frame, the frame segment is discretized into at least one event atom, wherein each event atom represents a coherent action sequence; for each of the event atom, the visual features of the continuous image frames corresponding to the event atom are extracted, and the visual features are converted into corresponding semantic descriptions to obtain a summary text of the event atom; the summary texts of each of the event atoms are spliced ​​in sequence according to the order in which the event atoms occur to obtain the initial video summary of the frame segment.

4. The method according to claim 2, characterized in that The video summary generation model includes: a long short-term memory network, a feedforward neural network, and an output network. The video summary generation model is used to analyze the decoding output result to obtain a multi-granularity video summary of the video question data, including: Analyzing the decoding output results using a long short-term memory network in the video summary generation model to obtain a plurality of candidate multi-granularity video summaries; Using a feedforward neural network in the video summary generation model to evaluate each of the candidate multi-granularity video summaries, and obtaining a confidence score for each of the candidate multi-granularity video summaries, wherein the confidence score is comprehensively determined based on features of the candidate multi-granularity video summaries and their relevance to the video question data; The output network in the video summary generation model is used to output the candidate multi-granularity video summary with the highest confidence, so as to obtain the multi-granularity video summary of the video question data.

5. The method according to claim 1, characterized in that The multimodal question data is analyzed using a question classification model to obtain corresponding question types, including: When the multimodal question data only includes multiple other modal question data in addition to the speech type voice question data, the multiple feature extraction networks in the question classification model are used to perform feature extraction on the multiple other modal question data respectively to obtain feature maps of each other modality; the feature fusion unit in the question classification model is used to fuse the feature maps of each other modality to obtain a corresponding fused feature map; the deep feature extraction unit and the adaptive context convolution unit in the question classification model are used in turn to perform feature extraction on the fused feature map to obtain a multimodal feature vector containing deep features and context information; the output unit in the question classification model is used to analyze the multimodal feature vector to obtain the probability distribution of the multimodal feature vector belonging to various question categories, and the question category with the highest probability is taken as the first question type corresponding to the multimodal question data.

6. The method according to claim 5, characterized in that Analyzing the multimodal question data using a question classification model to obtain corresponding question types also includes: When the multimodal question data includes both speech question data of the speech type and multiple other modal question data, the Mel-frequency cepstral coefficient features, high-order cumulative features and Mel-frequency cepstral coefficient distribution features of the speech question data are extracted; the Mel-frequency cepstral coefficient features, high-order cumulative features and Mel-frequency cepstral coefficient distribution features of the speech question data are respectively input into the three channels of the question classification model to obtain the feature vector output by each channel, wherein the types of the channels include: convolutional neural network, long short-term memory network, and bidirectional long short-term memory network; the feature vectors output by each channel are fused using the feature fusion unit in the question classification model to obtain a speech fusion feature vector; the output unit in the question classification model is used to analyze the speech fusion feature vector to obtain the probability distribution of the speech fusion feature vector belonging to various question categories, and the question category with the highest probability is taken as the second question type corresponding to the speech question data; Determining whether the first question type and the second question type are the same; When the first question type and the second question type are the same, using one of the first question type and the second question type as the question type corresponding to the multimodal question number; When the first question type and the second question type are different, a question type with the highest question priority is selected from the first question type and the second question type according to a preset question priority rule as the question type corresponding to the multimodal question number.

7. The method according to claim 1, characterized in that The emotional tendency recognition model includes: a single-modal feature extraction unit and a single-modal feature fusion unit corresponding to each of the multiple modal types, a feature splicing unit, and an emotional classification unit, wherein: Each of the unimodal feature extraction units is configured to extract a unimodal feature vector of the problem data of a corresponding modal type, and to extract a joint feature vector between the unimodal feature vector corresponding to the corresponding modal type and the unimodal feature vectors corresponding to other modal types; Each of the unimodal feature fusion units is configured to determine weight coefficients of the unimodal feature vectors and the multiple joint feature vectors output by the unimodal feature extraction unit of the corresponding modality type using an attention mechanism, and to perform weighted fusion on the unimodal feature vectors and the multiple joint feature vectors output by the unimodal feature extraction unit of the corresponding modality type using the weight coefficients to obtain a corresponding unimodal fusion feature vector; The feature splicing unit is used to splice the single-modal fusion feature vectors output by the single-modal feature fusion units corresponding to each of the modal types to obtain a multi-modal fusion feature vector; The emotion classification unit is used to determine the probability distribution of the multimodal fusion feature vector belonging to various emotion categories, and output the emotion category with the highest probability as the prediction result.

8. A work order processing method, characterized in that: Applicable to servers, including: Receive multiple work order information packets sent by multiple clients within a preset time period, wherein each work order information packet includes a sentiment tendency, a problem type, a pending work order, and a corresponding generation timestamp corresponding to the multimodal problem data reported by the corresponding client; Parsing each of the work order information packets, and comprehensively determining the processing priority of each of the work orders to be processed based on the sentiment tendency, problem type and generation timestamp of the work order to be processed in the parsing results; In the order of the processing priority from high to low, the preset large language model is called to process each of the pending work orders in turn, to obtain the problem result of each of the pending work orders, and each of the problem results is fed back to the corresponding client in turn.

9. A work order management system, characterized in that: The work order management system includes: multiple clients and a server, wherein: Each of the clients is configured to perform text extraction on the multimodal question data reported by the user to obtain corresponding target text data; analyze the multimodal question data using a question classification model to obtain a corresponding question type, and input the target text data into a work order template that matches the question type to obtain a work order to be processed, and determine a generation timestamp of the work order to be processed; analyze the multimodal question data using a sentiment tendency recognition model to obtain a corresponding sentiment tendency; and package the sentiment tendency, question type, work order to be processed, and corresponding generation timestamp corresponding to the multimodal question data into a work order information package and send it to the server; The server is used to receive multiple work order information packets sent by the multiple clients within a preset time period, wherein each of the work order information packets includes the sentiment tendency, problem type, pending work order and corresponding generation timestamp corresponding to the multimodal problem data reported by the corresponding client; parse each of the work order information packets, and comprehensively determine the processing priority of each of the pending work orders based on the sentiment tendency, problem type and generation timestamp of the pending work order in the parsing result; call the preset large language model to process each of the pending work orders in turn in order of the processing priority from high to low, obtain the problem result of each of the pending work orders, and feed back each of the problem results to the corresponding client in turn.

10. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, it implements the work order generation method described in any one of claims 1 to 7 or the work order processing method described in claim 8.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the work order generating method described in any one of claims 1 to 7 or the work order processing method described in claim 8 through the computer program.

Images