A two-dimensional quality inspection modeling method, device and storage medium based on text frames

Abstract

The application discloses a two-dimensional quality inspection modeling method and device based on a text frame and a storage medium, and comprises the following steps: obtaining service audio, pre-processing the service audio to obtain a time-domain continuous text frame sequence; labeling a service item related sentence corpus and training an item related sentence identification model; identifying and extracting features of each frame of text according to the item related sentence identification model to construct a frame feature vector and a frame feature matrix of each item; and designing and training a text frame quality inspection model for compliance quality inspection of a service work order. The time-domain continuous text frame sequence and the item related sentence are obtained, features of each frame of text and the item related sentence text are extracted and fused to obtain a frame feature vector, a time-domain continuous frame feature vector sequence forms a frame feature matrix containing service text in two dimensions of time and frame text space, a two-dimensional quality inspection model is designed to learn language and behavior features in a service process, and thus, abnormal conditions of a service work order can be more accurately identified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a two-dimensional quality inspection modeling method based on text frames, equipment and storage medium. BACKGROUND

[0002] Service exception identification is an important part of home service quality inspection. In the process of home service, due to the complexity of service scene and the uncontrollability of service process, various abnormal work orders are generated. At the same time, the particularity of home scene makes the application of supervision technology have many limitations, which makes it difficult to realize the compliance quality inspection of abnormal work orders. The existing quality inspection means mainly discovers abnormal work orders through artificial sampling service audio and service personnel reporting, which has problems such as excessive dependence on manual work, lack of quality inspection standard, low abnormal identification rate and so on. Therefore, how to better identify and inspect abnormal service work orders is a challenge faced by realizing the whole process supervision of home service, ensuring service compliance and qualification. SUMMARY

[0003] The technical problem to be solved by the present application is that in the process of home service quality inspection, the identification of abnormal work orders is excessively dependent on manual work, lacks quality inspection standard, and has low abnormal identification rate. The purpose is to provide a two-dimensional quality inspection modeling method based on text frames, equipment and storage medium, generate a time-continuous text frame sequence by uniform sampling, extract and fuse the features of each text frame to obtain a frame feature vector, and the time-continuous frame feature vector sequence constitutes a frame feature matrix containing service text in time and frame text space two-dimensional information. Design a two-dimensional quality inspection model, i.e. a text frame quality inspection model to learn the language and behavior characteristics of service personnel in the service process, so as to more accurately identify the compliance and abnormal situation of work orders.

[0004] The present application is realized by the following technical scheme:

[0005] The present application provides a two-dimensional quality inspection modeling method based on text frames in the first aspect, comprising the following steps:

[0006] Obtaining service audio, pre-processing the service audio to obtain a time-continuous text frame sequence;

[0007] Annotating the service project related sentence corpus, and training a project related sentence identification model;

[0008] According to the project related sentence identification model, identify and extract features of each text frame, and construct a frame feature vector and a frame feature matrix for each project;

[0009] Design and train a text frame quality inspection model for compliance quality inspection of service work orders.

[0010] The application generates a time-continuous text frame sequence by uniformly sampling the service text in the time domain, trains a project-related sentence recognition model to mark the project-related sentences in the frame text, extracts and fuses the features of each frame text and project-related sentence text to obtain a frame feature vector, and a time-continuous frame feature vector sequence constitutes an input feature matrix of the quality inspection model, which contains information of the service text in the time and frame text space two dimensions, so that the designed text frame quality inspection model can better learn the language and behavior characteristics of the service personnel in the service process, thereby more accurately identifying the compliance and abnormal conditions of the work order.

[0011] Further, the pre-processing of the service audio specifically includes:

[0012] performing ASR speech recognition on the service audio to obtain service text with a timestamp;

[0013] setting a sampling time parameter to uniformly sample the service text in the time dimension to obtain a continuous text frame sequence, which can be represented as:

[0014]

[0015] The sampling time is the main parameter for generating the text frame, and the selection of the sampling time needs to be considered comprehensively according to the actual business scenario. The time can be longer in a scenario with long service time and low corpus density, and the time can be shorter in a scenario with short service time or high corpus density. In theory, the shorter the sampling time, the more text frames and the richer the information expressed.

[0016] Further, the training of the project-related sentence recognition model includes:

[0017] obtaining historical work order corpus data, annotating service project-related sentence samples and negative samples;

[0018] selecting a suitable multi-label classifier according to the actual needs of quality inspection business;

[0019] training the classification model with the annotated samples to obtain a project-related sentence recognition model for text frame feature extraction.

[0020] The project-related sentence recognition is to obtain the project related to the input sentence. A sentence can be related to one project or multiple projects, so a multi-label classification model is selected to implement it. The role of the model is to mark the project-related sentences of each frame text in the process of constructing the frame feature vector. The project-related sentence information is extracted through the feature extraction to constitute a sub-vector of the frame feature vector in the service project dimension.

[0021] Further, the frame feature vector construction step specifically includes:

[0022] Each frame of text is segmented, and all punctuation marks are used as a division symbol;

[0023] According to the project-related sentence identification model, each sentence in the text frame is identified to obtain the relevant sentence information of each project in the frame text.

[0024] The frame text features and the relevant sentence features are extracted and fused to form a frame feature vector of each project in the frame text, which can be represented as:

[0025]

[0026] Further, the feature extraction and fusion method specifically includes:

[0027] The feature information of the frame text in the word and sentence dimensions is extracted, including: word number, sentence number, project-related sentence number, relevant sentence proportion, and relevant sentence word proportion;

[0028] The word and sentence dimension features are fused by concatenation to form a frame feature vector.

[0029] Further, the construction of the frame feature matrix specifically includes:

[0030] All frame feature vectors of the service text are arranged and combined in time sequence to obtain a frame feature matrix of the corpus;

[0031] The frame feature matrix contains information in the time and frame text space dimensions of the service corpus.

[0032] Further, the design of the text frame quality inspection model specifically includes:

[0033] The model input layer is constructed according to the frame feature matrix;

[0034] The convolution kernel is determined according to the frame feature vector, and the column vector is obtained according to the convolution kernel;

[0035] The column vector is passed through the max-pooling layer to obtain a feature representation representing the local frame of the text;

[0036] The pooled feature vector is input into the fully connected neural network to obtain the final output of the quality inspection model.

[0037] Further, the training of the text frame quality inspection model specifically includes:

[0038] The actual quality inspection target is obtained, and the output of the quality inspection model is determined.

[0039] The historical work order service project corpus is obtained, and the training samples are labeled according to the quality inspection target;

[0040] The designed text frame quality inspection model is trained using the labeled samples;

[0041] The second aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement a two-dimensional quality inspection modeling method based on a text frame when executing the program.

[0042] The third aspect of the present application provides a computer readable storage medium, which stores a computer program, wherein the program is configured to implement a two-dimensional quality inspection modeling method based on a text frame when executed by a processor.

[0043] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0044] 1. The service corpus is expanded from one-dimensional text space to two-dimensional time-frame text space by the text frame processing method, which more comprehensively reflects the language and behavior characteristics in the service process and provides a basis for the construction of the two-dimensional quality inspection model.

[0045] 2. The frame corpus, service-related corpus and irrelevant corpus word-level and sentence-level features are fused in the construction process of the frame feature vector, so that the information representation of the feature vector is more rich.

[0046] 3. The two-dimensional quality inspection model is constructed by using CNN plus full connection neural network, which can learn the language and behavior characteristics from local to global in the service process and improve the reliability and accuracy of the quality inspection result. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor. In the drawings:

[0048] Figure 1 The flowchart in the embodiments of the present application;

[0049] Figure 2 The project-related sentence classification model schematic diagram in the embodiments of the present application;

[0050] Figure 3 The text frame feature extraction flowchart in the embodiments of the present application;

[0051] Figure 4 The text frame quality inspection model design diagram in the embodiments of the present application. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given below in combination with embodiments and drawings. The illustrative embodiments and their descriptions are only used to explain the present application and do not limit the present application.

[0053] Embodiment 1

[0054] As shown in the embodiment, the first aspect of the embodiment provides a two-dimensional quality inspection modeling method based on text frames, comprising the following steps: Figure 1

[0055] obtaining service audio, pre-processing the service audio to obtain a time-continuous text frame sequence;

[0056] annotating service item related sentences, and training an item related sentence recognition model;

[0057] recognizing and extracting features of each frame of text according to the item related sentence recognition model, constructing a frame feature vector and a frame feature matrix of each item;

[0058] designing and training a text frame quality inspection model for compliance quality inspection of service work orders.

[0059] The embodiment uniformly samples the service text in the time domain to generate a time-continuous text frame sequence, trains an item related sentence recognition model to mark the item related sentences in the frame text, extracts and fuses features of each frame of text and the item related sentence text to obtain a frame feature vector, and a time-continuous frame feature vector sequence constitutes a frame feature matrix containing information of the service text in the time and frame text space two dimensions, and a two-dimensional quality inspection model, i.e., a text frame quality inspection model, learns language and behavior characteristics of service personnel in the service process, so as to more accurately identify compliance and abnormal conditions of work orders.

[0060] In some possible embodiments, the pre-processing of the service audio specifically comprises:

[0061] performing ASR speech recognition on the service audio to obtain service text with time stamps;

[0062] setting a sampling time parameter, uniformly sampling the service text in the time dimension to obtain a continuous text frame sequence:

[0063]

[0064] ​In some possible embodiments, the parameter of the text frame is mainly the sampling time, and the selection of the sampling time needs to be comprehensively considered according to the complexity of the actual service and the quality inspection demand. The time can be longer in a scenario with long service time and low corpus density, and the time can be shorter in a scenario with short service time or high corpus density. In theory, the shorter the sampling time is, the more information the text frame can express, but in order to extract more semantic features, a complete sentence is needed in the text frame, so the sampling time cannot be too small. Moreover, in the actual application process, the text needs to be generated by ASR, so the text frame sampling time needs to be combined with the ASR input duration.

[0065] In some possible embodiments, training the project-related sentence identification model comprises:

[0066] Obtaining historical work order corpus data, annotating service project-related sentence samples and negative samples;

[0067] The annotation of the project-related sentence samples is preferably selected from the text data generated in the real service, and the number of samples of each service project is preferably balanced. The selection range of the irrelevant samples can be larger, and in addition to the corpus generated in the real service, the corpus data in other similar service scenarios can also be selected.

[0068] According to the actual demand of the quality inspection business, a suitable multi-label classifier is selected;

[0069] The principle of project-related sentence identification is to obtain the related project through model algorithm according to the input corpus information. A sentence can be related to one project or multiple projects. Therefore, a multi-label classification model is selected to realize it. The traditional classification model based on machine learning or the classification model based on neural network and deep learning can be selected. The accuracy and running speed of the classifier are two performance indicators that need to be considered. The classifier based on deep learning has the highest accuracy, but the running speed is relatively slow and the project landing cost is relatively high. Therefore, under the premise of meeting the needs of the project, the machine learning and neural network classifier is preferred, such as SVM (Support Vector Machine), MLP (Neural Network Model), Fasttext, etc. are good classifiers with comprehensive performance, as shown in Figure 2 .

[0070] The classification model is trained with the annotated samples to obtain a project-related sentence identification model for text frame feature extraction.

[0071] In some possible embodiments, constructing the feature vector specifically comprises:

[0072] Each frame of text is divided into sentences, and all punctuation marks are used as the division symbol;

[0073] According to the project-related sentence identification model, each sentence corpus in the text frame is identified to obtain the related sentence information of the frame text about each project.

[0074] Feature extraction is performed on each frame of text data corresponding to each quality inspection item. In the process of feature extraction, a relevant sentence classification model is used to identify each sentence corpus in the text frame to determine whether it is related to a certain item. Finally, the information of each item in the word and sentence dimensions is statistically summarized according to the relevant sentences of the item.

[0075] Frame text features and relevant sentence features are extracted and fused to form frame feature vectors of each item on the frame text.

[0076] In some possible embodiments, the feature extraction and fusion method specifically includes:

[0077] The feature information of the frame text in the word and sentence dimensions is extracted, including: the number of words, the number of sentences, the number of relevant sentences, the proportion of relevant sentences, and the proportion of relevant sentence words;

[0078] The features in the word and sentence dimensions are fused by concatenation to form the frame feature vector. The process of obtaining the frame feature vector from the text frame can be represented as:

[0079]

[0080] A feature vector representing the entire work order can be constructed, or a feature vector for each service item to be quality inspected can be constructed. The present application adopts the scheme of constructing a feature vector for each quality inspection item.

[0081] In some possible embodiments, constructing the frame feature matrix specifically includes:

[0082] All frame feature vectors of the service corpus are arranged and combined in time sequence to obtain the frame feature matrix of the corpus.

[0083] As shown in Figure 3 , wherein: the number of text frames T, the number of quality inspection items m, the length of the feature vector d, and finally m T*d feature sets should be output, i.e., the frame feature matrix of the quality inspection item on the service text.

[0084] The frame feature matrix contains information in two dimensions of time and frame text space. As shown in Figure 4 , the frame feature matrix of the text is obtained after feature extraction, and a CNN plus neural network classifier is used to realize service compliance quality inspection.

[0085] The designed quality inspection model has 4 layers:

[0086] 1. Input layer: the feature vector sequence composed of feature vectors of all text frames, i.e., the frame feature matrix, which can be represented as F 0:T-1 .

[0087] 2. Convolutional layer, multiple filters can be used according to different needs, the width of the convolution kernel must be the same as the frame feature vector, so the convolution process is up and down, here the convolution kernel with length d and height a is selected, the formula is as follows:

[0088]

[0089] Where x ki is the convolution result, W k ∈R dα and b k are the weight matrix and bias term of the kth convolution kernel, respectively, a k is the height of the kth convolution kernel, and F i:i+α-1 represents the frame feature vector sequence F i , F i+1 ,…, F i+α-1 , and the activation function is relu.

[0090] 3. Pooling layer, max-pooling layer for each convolution kernel, after convolution, a column vector X k is generated, and after max-pooling layer, a feature scalar h k is generated, which can represent the N-GRN result of the frame feature, representing the local frame feature abstraction of the text, the formula is as follows:

[0091] h k = max(X k )

[0092] 4. Fully connected layer, the pooled feature vector is taken as input, and the global modeling capability of the fully connected neural network is used to obtain the final quality inspection result y n , the formula is as follows:

[0093] y n = sigmoid(W n H+b n )

[0094] Where W n = [w n1 , w n2 ,…, w nk ] is the weight value corresponding to each output, b n is the bias term, H = [h1, h2, …, h k ] is the pooled feature vector, and the activation function is sigmoid.

[0095] In some possible embodiments, training the text frame quality inspection model specifically includes:

[0096] Obtaining an actual quality inspection target and determining a quality inspection model output;

[0097] Before designing the quality inspection model, the output of the model, i.e. the target of service quality inspection, needs to be determined first. The quality inspection target can be one or more, for example: whether the service is compliant, whether there is cheating behavior, service level determination, etc. The total number of quality inspection targets should be determined according to the complexity of the home service scene and the specific quality inspection requirements. Each quality inspection target corresponds to an output of the model. As shown in Table 1, the quality inspection model output table:

[0098] Table 1 Quality inspection model output table

[0099] Quality inspection target Model output Quality inspection target 1 y1 Quality inspection target 2 y2 … … Quality inspection target n yn

[0100] Obtain historical work order service project corpus, and label training samples according to quality inspection targets;

[0101] After determining the quality inspection targets, the training samples need to be labeled according to the model output list. A model that meets production requirements requires a certain number of training samples. According to model training and testing experience: a model with n quality inspection targets requires about n*10,000 training samples. The more quality inspection targets, the more information the model outputs, and the more training samples are needed. The cost of the project also increases, so the number of model output bits should be controlled in the appropriate range during the early design of the model. The labeling of training samples is preferably selected from the text data generated in real service, and the number of positive samples corresponding to each output bit is kept balanced as much as possible. In addition to the corpus generated by real service, corpus data from other similar service scenarios can also be selected.

[0102] Train the designed text frame quality inspection model with labeled samples;

[0103] After completing the model training corpus labeling, text frame feature engineering construction and model parameter selection, the text frame quality inspection model is obtained after training the model. Modeling and training can be done on many mature open source frameworks, such as pytorch, tensorflow, sk-learn, etc.

[0104] The implementation process of the home service process quality inspection system based on text frame quality inspection is as follows:

[0105] 1. The audio equipment worn by the home service personnel collects service speech and uploads it to the quality inspection server.

[0106] 2. The service speech is converted by ASR to obtain time-stamped service corpus.

[0107] 3. Sample frame and feature extraction are performed on the service corpus to obtain a text frame feature matrix.

[0108] 4. Frame feature matrix is input into L quality inspection models of the to-be-inspected items, to obtain the quality inspection result of each service item. (Note: assuming L to-be-inspected items, the quality inspection model of each item is independent and needs to be trained in advance)

[0109] 5. The quality inspection result is pushed to the service supervision system, and the quality inspection of the service work order is completed.

[0110] The second aspect of the embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor is configured to implement a two-dimensional quality inspection modeling method based on a text frame when executing the program.

[0111] The third aspect of the embodiment provides a computer readable storage medium having a computer program stored thereon, and the program is configured to implement a two-dimensional quality inspection modeling method based on a text frame when executed by a processor.

[0112] Embodiment 2

[0113] The embodiment provides a two-dimensional quality inspection modeling method in a specific service scenario.

[0114] Step 1: Set the text frame sampling time. Since all texts need to be generated by ASR, and the transcription step of ASR is 1 minute, the text frame sampling time t is set to 1 minute.

[0115] Step 2: Determine the quality inspection target. Here, three common regulatory requirements are taken: service qualification, cheating detection, and noise detection,

[0116] Step 3: Corpus annotation. There are two parts, service item compliance annotation for service quality inspection model training, and service item related sentence annotation for service item related sentence identification. The annotation of the training sample selects the text data generated in the real service, and the number of positive samples corresponding to each output bit is kept balanced as much as possible. The model that meets the production requirements needs a certain number of training samples. Here, 30,000 work orders are selected for training sample annotation.

[0117] Step 4: Project-related sentence identification model training. Project-related sentence identification is to obtain which projects it is related to through model algorithm according to the input corpus information. A sentence can be related to one project or multiple projects. A multi-label classification model is selected to achieve this, and machine learning and neural network classifiers are preferred in terms of landing cost and performance. In this embodiment, SVM (Support Vector Machine) is used as a classifier, and the sk-learn framework is used for modeling and training.

[0118] Step 5: Frame feature vector construction. In the process of feature extraction, a relevant sentence classification model is used to identify each sentence in the text frame and determine whether it is related to a certain project. Finally, the information of each project word dimension and sentence dimension is summarized according to the relevant sentences of the project to form the feature vector of the project in this frame text. The information for constructing the feature vector in this embodiment is: [frame word number, frame sentence number, frame average sentence length, project related word number, project related sentence number, project related average sentence length, project unrelated word number, project unrelated sentence number, project unrelated average sentence length, project related sentence number / frame word number, project related sentence number / frame sentence number, and project related average sentence length / frame average sentence length], and the vector dimension is 12.

[0119] Step 6: Determine the text frame quality inspection model parameters. Mainly the setting of the convolution kernel height and the number of kernels. Take the convolution kernel height α as 2, 3, 4, and 5 respectively, and each has 16 kernels, a total of 64 convolution kernels.

[0120] Step 7: Model training of the text frame quality inspection model. After completing the work order corpus annotation, text frame feature engineering construction, and model parameter selection, use the pytorch framework to model and train to obtain the text frame quality inspection model. At this point, the quality inspection model training is completed, and the model will be applied to the quality inspection project next.

[0121] Step 8: The audio device worn by the home service personnel collects service speech and uploads it to the quality inspection server. The service speech is converted by ASR to obtain the timestamped service audio.

[0122] Step 9: Sampling and framing of the service audio, feature extraction, and obtaining the text frame feature matrix of the to-be-inspected project A. The feature matrix is input into the quality inspection model of project A to obtain the quality inspection result, for example:

[0123] {Project A: {Service eligible: 1, cheating detection: 0, noise detection: 0}}

[0124] Step 10: According to the quality inspection result of each service project, the compliance of the entire work order is identified. The compliance judgment logic in this example is: at least 1 service eligible, no cheating, and no noise project is identified. Therefore, this work order is determined to be compliant.

[0125] Step 11: Push the quality inspection result to the business assessment system: {Work order status: compliant, containing project: {Project A}}, and the service work order quality inspection is completed this time.

[0126] The above specific embodiments further illustrate the purpose, technical solutions, and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and does not limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A two-dimensional quality inspection modeling method based on text frames, characterized in that, Includes the following steps: The service audio is acquired, and the service audio is preprocessed to obtain a temporally continuous sequence of text frames. Annotate the corpus of service-related sentences and train a sentence recognition model for service-related sentences. Based on the project-related sentence recognition model, each frame of text is identified and its features are extracted to construct the frame feature vector and frame feature matrix for each project; The steps for constructing the frame feature vector specifically include: segmenting each frame of text into sentences, using all punctuation marks as segmentation delimiters; identifying each sentence in the text frame using a project-related sentence recognition model to obtain relevant sentence information about each project in the frame text; extracting and fusing the frame text features and relevant sentence features to form the frame feature vector for each project in the frame text; the extraction and fusing of frame text features and relevant sentence features specifically includes: extracting feature information of the frame text in the character and sentence dimensions, including: number of characters, number of sentences, number of project-related sentences, proportion of related sentences, and proportion of related sentence characters; the character and sentence dimension features are fused together to form the frame feature vector; The specific steps for constructing the frame feature matrix include: arranging and combining all frame feature vectors of the service text in temporal order to obtain the frame feature matrix of the corpus; the frame feature matrix contains feature information of the service corpus in both time and frame text space dimensions, and is the input of the two-dimensional quality inspection model; Design and train a text frame quality inspection model for compliance inspection of service work orders. The design of the text frame quality inspection model specifically includes: constructing the model input layer based on the frame feature matrix; determining the convolution kernel based on the frame feature vector, and obtaining the convolution column vector based on the convolution kernel; passing the convolution column vector through a max pooling layer to obtain the feature representation of the local text frame; and inputting the pooled feature vector into a fully connected neural network to obtain the final output of the quality inspection model.

2. The two-dimensional quality inspection modeling method based on text frames according to claim 1, characterized in that, The preprocessing of the service audio specifically includes: Perform ASR speech recognition on the service audio to obtain the service text with a timestamp; By setting the sampling time parameter, the service text is sampled uniformly in the time dimension to obtain a continuous text frame sequence.

3. The two-dimensional quality inspection modeling method based on text frames according to claim 1, characterized in that, The training project-related sentence recognition model includes: Obtain historical work order corpus data and label service item-related sentence samples and negative samples; Select a suitable multi-label classifier based on the actual needs of quality inspection operations; The classification model is trained using labeled samples to obtain an item-related sentence recognition model, which is used for text frame feature extraction.

4. The two-dimensional quality inspection modeling method based on text frames according to claim 1, characterized in that, The trained text frame quality inspection model specifically includes: Obtain the actual quality inspection targets and determine the output of the quality inspection model; Obtain historical work order service item corpus and label training samples according to quality inspection objectives; The text frame quality inspection model was trained using labeled samples.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a two-dimensional quality inspection modeling method based on text frames as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a two-dimensional quality inspection modeling method based on text frames as described in claims 1 to 4.

Images