A method and device for compliance review of operation processes based on speech recognition

Abstract

The present invention provides a method and device for compliance review of an operation process based on speech recognition. The method includes: performing recognition and conversion on speech data collected in a target operation scenario to obtain target text data; inputting the target text data into a pre-trained language model to output several groups of entities, events, associations between entities and entities and / or events, and associations between events; according to the several groups of entities, events and associations, using a target operation process template corresponding to the target operation scenario to obtain an operation process to be reviewed; performing compliance review on the operation process to be reviewed to obtain a review result. This solution realizes automated, intelligent, and highly adaptable compliance review of the entire process of voice conversations in operations, with high review coverage and applicability to complex and variable scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and particularly to a method and device for compliance review of operation processes based on speech recognition. Background Art

[0002] In operation scenarios of many industries such as logistics, mining, railways, aviation, and industry, the whole-process communication between on-site operators and dispatching and commanding personnel mainly relies on voice communication methods such as walkie-talkies, radios, and mobile phones. To ensure operation safety and quality, a set of standardized inspection processes need to be carried out before many actions are implemented. Therefore, these voice conversations need to comply with relevant operation specifications and requirements.

[0003] Currently, the compliance inspection of operation voice conversations mainly includes manual sampling method, keyword matching method, and rule engine method. Among them, the manual sampling method extracts some voice recordings manually for manual listening and evaluation, but it depends on manual operation, has a large workload and low efficiency, and it is difficult to achieve full coverage; the keyword matching method converts the voice into text and then matches it through a preset keyword library to examine whether there are any illegal terms. Although this method can achieve automated inspection, its semantic understanding ability in complex scenarios is limited and misjudgments are likely to occur; the rule engine method analyzes and conducts compliance review on the text after voice conversion through preset business rules and logics. Although this method can handle certain complex logics, the formulation and maintenance of the rules are relatively complex and lack flexibility, and it is difficult to handle illegal situations beyond the preset scope and difficult to cope with changing scenarios. Therefore, a method and device for compliance review of operation processes based on speech recognition are needed. Summary of the Invention

[0004] The present invention provides a method and device for compliance review of operation processes based on speech recognition, which realizes automated, intelligent, and highly adaptable compliance review of the whole-process operation voice conversations, and has a high review coverage rate and is applicable to complex and changeable scenarios.

[0005] In a first aspect, the present invention provides a method for compliance review of operation processes based on speech recognition, including:

[0006] Performing recognition conversion on the voice data collected in the target operation scenario to obtain target text data;

[0007] Inputting the target text data into a pre-trained language model, and outputting several groups of entities, events, the relationships between entities and entities and / or events, and the relationships between events;

[0008] According to several groups of the entities, the events, and the relationships, obtaining an operation process to be reviewed by using a target operation process template corresponding to the target operation scenario;

[0009] Perform a compliance review on the to-be-reviewed operation process to obtain a review result.

[0010] Optionally, the recognition and conversion of the voice data collected in the target operation scenario to obtain target text data includes:

[0011] Preprocess the voice data to obtain preprocessed voice data;

[0012] Mark the time stamp and location information for the preprocessed voice data;

[0013] Perform recognition and conversion on the preprocessed voice data to obtain temporary text data;

[0014] Correct and semantically error-check the temporary text data according to the dialect knowledge base to obtain initial text data;

[0015] Sort the initial text data in chronological order of the time stamps and correspondingly add the location information to the initial text data to obtain the target text data.

[0016] Optionally, the pre-trained language model is trained by the following method:

[0017] Obtain the training parameters of the language model and freeze them; wherein, the language model is used to identify the operation full-process text data in the historical text data in the target operation scenario; the operation full-process text data includes several groups of entities, events, and association relationships;

[0018] Construct a low-rank matrix in the architecture of the pre-trained language model to obtain a fine-tuned language model;

[0019] Train the fine-tuned language model with at least two groups of first sample sets to obtain the pre-trained language model; wherein, each group of the first sample sets includes the historical text data in the target operation scenario as the input and several groups of entities, events, and association relationships of the historical text data as the output.

[0020] Optionally, the language model is trained by the following method:

[0021] Analyze the historical text data in the target operation scenario to obtain the operation full-process text data in the target operation scenario;

[0022] The language model is trained with at least two groups of second sample sets; wherein, each group of the second sample sets includes the historical text data as the input and the operation full-process text data of the historical text data as the output.

[0023] Optionally, the entities include the department to which the staff member belongs, the staff member, the equipment name, the work area, the geographical location of the work, and the working hours; the events include the actions and current status of performing any operation task; the actions include completing inspections, waiting for commands, starting construction, and completing construction; the current status includes: all staff members are present, the equipment is being inspected, the equipment is ready, and the equipment has completed the operation; the association relationships include the hierarchical relationships between the work departments to which the staff members belong and the sequence order between the operation tasks.

[0024] Optionally, based on several groups of the entities, the events, and the association relationships, obtaining the operation process to be reviewed by using the target operation process template corresponding to the target operation scenario includes:

[0025] Obtain the target operation process template adopted by the target operation scenario; wherein, different operation scenarios adopt different operation process templates; the target operation process template is used to output target entities, target events, and target association relationships in a specified output format;

[0026] Determine several groups of the target entities, the target events, and the target association relationships from several groups of the entities, the events, and the association relationships according to the specified output format;

[0027] Output several groups of the target entities, the target events, and the target association relationships in the specified output format and chronological order to obtain the operation process to be reviewed.

[0028] Optionally, performing compliance review on the operation process to be reviewed to obtain a review result includes:

[0029] Obtain the standard operation process under the target operation scenario;

[0030] Compare the standard operation process with the operation process to be reviewed to determine a review result including non-compliant process nodes and potential risk process nodes.

[0031] Optionally, comparing the standard operation process with the operation process to be reviewed to determine a review result including non-compliant process nodes and potential risk process nodes includes:

[0032] Segment the operation process to be reviewed to obtain a first type of operation process corresponding to the standard operation process and a second type of operation process not corresponding to the standard operation process;

[0033] Review the first type of operation process according to the standard operation process to determine non-compliant process nodes;

[0034] Identify each of the second type of operation processes as a potential risk process node;

[0035] Conduct a risk assessment on the non - compliant process nodes and the potential risk process nodes to obtain a risk score, and determine a solution based on the risk score.

[0036] In a second aspect, the present invention provides an operation process compliance review device based on speech recognition, including:

[0037] A speech recognition module for identifying and converting the speech data collected in the target operation scenario to obtain target text data;

[0038] An analysis module for inputting the target text data into a pre - trained language model, outputting several groups of entities, events, the relationships between entities and entities and / or events, and between events; and obtaining the operation process to be reviewed by using the target operation process template corresponding to the target operation scenario according to the several groups of the entities, the events, and the relationships;

[0039] A review module for conducting a compliance review on the operation process to be reviewed to obtain a review result.

[0040] In a third aspect, an embodiment of the present invention further provides a computing device, including a memory and a processor. When the processor executes the computer program stored in the memory, the method described in any first aspect of this specification is implemented.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer - readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the method described in any first aspect of this specification.

[0042] In a fifth aspect, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the method described in any first aspect of this specification are implemented.

[0043] An embodiment of the present invention provides a method and apparatus for compliance review of operation processes based on speech recognition. By recognizing and converting the speech data collected in a target operation scenario, target text data is obtained. Then, the target text data is input into a pre-trained language model, and key information in the target operation scenario is output, including entities, events, the relationships between entities and entities and / or events, and the relationships between events. Then, a to-be-reviewed operation process is output using the target operation process template in the target operation scenario. Finally, a compliance review is performed on the to-be-reviewed operation process to obtain a review result. In this way, this solution utilizes a pre-trained language model with powerful semantic understanding ability, without designing extraction rules, and can determine entities, events, and their relationships only based on concepts, attributes, etc. in the target text data. At the same time, through a specific target operation process template, operation process information is accurately extracted, thus breaking through the limitations of traditional keyword matching, achieving a deep understanding of complex operation scenarios, and further improving the accuracy and flexibility of compliance review. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0045] Figure 1 is a flowchart of a method for compliance review of operation processes based on speech recognition provided by an embodiment of the present invention;

[0046] Figure 2 is a hardware architecture diagram of a computing device provided by an embodiment of the present invention;

[0047] Figure 3 is a structural schematic diagram of an apparatus for compliance review of operation processes based on speech recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0049] The following describes the specific implementation manners of the concept of this application.

[0050] Please refer toFigure 1 , an embodiment of the present invention provides a method for compliance review of an operation process based on speech recognition, and the method includes:

[0051] Step 100: Identify and convert the speech data collected in the target operation scenario to obtain target text data;

[0052] Step 102: Input the target text data into a pre-trained language model, and output several groups of entities, events, the associations between entities and entities and / or events, and the associations between events;

[0053] Step 104: According to several groups of entities, events and associations, use the target operation process template corresponding to the target operation scenario to obtain the operation process to be reviewed;

[0054] Step 106: Conduct a compliance review on the operation process to be reviewed to obtain a review result.

[0055] In the embodiment of the present invention, by identifying and converting the speech data collected in the target operation scenario, target text data is obtained, and then the target text data is input into a pre-trained language model to output the key information in the target operation scenario, including entities, events, the associations between entities and entities and / or events, and the associations between events, etc. Then, the target operation process template in the target operation scenario is used to output the operation process to be reviewed, and finally, by conducting a compliance review on the operation process to be reviewed, a review result is obtained. In this way, this solution uses a pre-trained language model with strong semantic understanding ability, without designing extraction rules, and can determine entities, events and their associations only based on concepts, attributes, etc. in the target text data. At the same time, through a specific target operation process template, the operation process information is accurately extracted, thereby breaking through the limitations of traditional keyword matching, realizing a deep understanding of complex operation scenarios, and further improving the accuracy and flexibility of compliance review.

[0056] The following describes Figure 1 the execution manners of the following steps.

[0057] First, in step 100, the speech data collected in the target operation scenario is identified and converted to obtain target text data, including:

[0058] Preprocess the speech data to obtain preprocessed speech data;

[0059] Mark the timestamp and location information for the preprocessed speech data;

[0060] Identify and convert the preprocessed speech data to obtain temporary text data;

[0061] Correct and semantically correct the temporary text data according to the dialect knowledge base to obtain the initial text data;

[0062] Sort the initial text data in the chronological order of the timestamps, and correspondingly add location information to the initial text data to obtain the target text data.

[0063] Specifically, the present invention preferably uses a highly sensitive microphone array or an intelligent intercom to capture the voice communication content, identity information, and location information among operators in real time in the target operation scenario. Among them, the location information is used to represent the geographical location of the current voice data in the target operation scenario. The preprocessing is used to improve the voice quality, including noise reduction, decibel enhancement, etc.

[0064] In the present invention, by marking the timestamps and location information for the voice data, the subsequent analysis efficiency and the operation process to be reviewed are improved. At the same time, since there are dialects in the actual operation scenario, it is also necessary to further use the dialect knowledge base to correct and semantically correct the temporary text data to improve the conversion accuracy from voice to text.

[0065] In a specific embodiment, step 100 can use a speech recognition model trained by a deep learning algorithm (such as the Transformer or Conformer architecture) and a dialect knowledge base to perform recognition and conversion on the voice data, specifically including: (1) constructing a dialect knowledge base: providing background information corresponding to the dialect and Mandarin for the speech recognition model to assist the correction process; (2) training the speech recognition model with dialect voices and their corresponding dialect texts, Mandarin voices, and corresponding Mandarin texts to obtain a trained speech recognition model; (3) inputting the voice data into the trained speech recognition model and outputting the target text data. In this way, the trained speech recognition model can recognize and label the dialects, providing a direction for correction; at the same time, it can also perform context understanding and correction. Through the powerful language understanding ability of the model, combined with the context and the dialect knowledge base, correct the transcription errors of the dialects and perform semantic error correction, thereby improving the conversion accuracy from voice data to text data.

[0066] For example, in the dialect knowledge base, the dialect is "Nong hao fa", and its corresponding Mandarin is "Ni hao ma"; the dialect is "Ge shi qing za nong", and its corresponding Mandarin is "Zhe ge shi qing zen me ban".

[0067] In step 102, the pre-trained language model is trained by the following method:

[0068] S1: Recognize and convert the historical voice data in the obtained target operation scenario to obtain historical text data; analyze the historical text data in the target operation scenario to obtain the full-process operation text data in the target operation scenario;

[0069] S2: Extract key information from the text data of the entire operation process to obtain several groups of entities, events, and association relationships in the target operation scenario;

[0070] S3: The language model is trained through at least two groups of second sample sets; wherein, each group of second sample sets includes historical text data as input and text data of the entire operation process of this historical text data as output

[0071] S4: Obtain and freeze the training parameters of the language model; wherein, the language model is used to identify the text data of the entire operation process in the historical text data of the target operation scenario; the text data of the entire operation process includes entities, events, and association relationships;

[0072] S5: Construct a low-rank matrix in the architecture of the pre-trained language model to obtain a fine-tuned language model;

[0073] S6: Train the fine-tuned language model with at least two groups of first sample sets to obtain a pre-trained language model; wherein, each group of first sample sets includes historical text data in the target operation scenario as input and several groups of entities, events, and association relationships of this historical text data as output.

[0074] It should be noted that each group of entities, events, and association relationships corresponds to a time node or time period, that is, the voice data under this time node or time period can be converted into a group of entities, events, and association relationships. Describing the voice data through the entity-event-association relationship group is beneficial to improving the efficiency and accuracy of obtaining the operation process.

[0075] Specifically, in step S1, the method of step 100 is used for identification and conversion to obtain historical text data, and the text data irrelevant to the operation process in the historical text data is removed to obtain the text data of the entire operation process. In step S2, the key information extraction adopts the sentence component analysis method to determine the subject, predicate, object, complement, attributive, adverbial, and appositive from the text data, and further determine the entities, events, and association relationships according to the determined sentence components. Entities are identified and determined from the sentence components according to predefined probabilities, such as people, equipment, locations, etc. The language model in step S3 is a large language model, such as GPT-3, LLaMA, etc.

[0076] Specifically, in steps S5 and S6, the low-rank matrix includes a first low-rank matrix and a second low-rank matrix, and for each layer of the language model architecture, the following is performed:

[0077] Introduce the first low-rank matrix and the second low-rank matrix in this layer, and determine the output of this layer according to the input of this layer, the training parameters of the language model, the first low-rank matrix, and the second low-rank matrix;

[0078] The fine-tuned language model introduced with a low-rank matrix is trained using at least two groups of first sample sets, and the corresponding first low-rank matrix and second low-rank matrix for each layer are determined to obtain a pre-trained language model;

[0079] wherein, the output of this layer is determined by the following formula: h = (W0 + BA)x

[0080] wherein, h is the output of this layer, x is the input of this layer, W0 is the training parameter of the language model; A is the first low-rank matrix; B is the second low-rank matrix. The matrix dimensions of A and B are smaller than those of W0.

[0081] Specifically, when training the fine-tuned language model, the original training parameter W0 is frozen, and low-rank approximation training is performed using matrices A and B with smaller parameter amounts. At the beginning of training, A is initialized with random Gaussian distribution, and B is initialized as a zero matrix, ensuring that training starts from W0. In this way, only A is trained for dimensionality reduction and B is trained for dimensionality increase during training; after training, only the parameters of low-rank matrices B and A need to be saved. By constructing low-rank matrices in the language model, the present invention greatly reduces the training cost, and at the same time can quickly respond to updates when updating the fine-tuned language model.

[0082] In a preferred embodiment, the entities include the department to which the staff belongs, the staff, the equipment name, the work area, the work geographical location, and the working time; the events include the actions of performing any operation task and the current status; the actions include completing inspection, waiting for commands, starting construction, and construction completion; the current status includes: all staff members are present, the equipment is being inspected, the equipment is ready, and the equipment has completed the operation; the association relationships include the hierarchical relationships between the work departments to which the staff belong and the sequence of operation tasks.

[0083] In the present invention, since the text data contains rich information on construction operation scenarios and involves multiple roles, various possible events and instructions, in order to collect the operation processes in the operation scenarios, it is necessary to extract key information from the text data of the entire operation process according to the standard operation process, operation specifications, safety requirement documents, etc., so as to facilitate determining the operation process from the text data. Specifically, for example, in the electrical construction operation scenario, the entities are the staff members: such as "construction team", the departments to which the staff members belong: such as "command center", "electrical inspection team", the equipment names: such as "equipment 031", "equipment 011", "equipment 052", the working geographical locations: such as "construction area", "electrical equipment inspection area", and the working time: in a standardized time format. The events are actions: such as "complete inspection", "wait for command", "issue command", "start construction", "construction completed", and the current statuses: such as "all staff members are present", "electrical equipment is being inspected", "electrical equipment is ready", "electrical equipment has completed the operation". The association relationships are superior-subordinate relationships: such as the "command center" issuing commands to the "construction team", the sequence: such as the electrical equipment inspection task is before the electrical equipment construction task, that is, the sequence in which events occur, and the execution relationship between the entity and the event: such as the electrical inspection team conducts electrical equipment inspection.

[0084] In the present invention, by constructing entity, event, and association relationship groups, the key information in the dialogue can be accurately extracted, the key information in the voice data can be automatically extracted without being restricted by the keyword library, it can adapt to different operation scenarios, achieve full coverage of the entire process of the operation process, avoid the omission problems of manual sampling inspection, and at the same time flexibly determine a more accurate operation process.

[0085] In step 104, according to several groups of entities, events, and association relationships, the operation process to be reviewed is obtained by using the target operation process template corresponding to the target operation scenario, including:

[0086] Obtain the target operation process template adopted by the target operation scenario; wherein, different operation scenarios adopt different operation process templates; the target operation process template is used to output the target entity, target event, and target association relationship according to the specified output format;

[0087] According to the specified output format, several groups of target entities, target events, and target association relationships are determined from several groups of entities, events, and association relationships;

[0088] Output several groups of target entities, target events, and target association relationships according to the specified output format and time sequence to obtain the operation process to be reviewed.

[0089] For example, for example, the specified output format of the operation process template is:

[0090] Event list:

[0091] {

[0092]

[0093]

[0094] In the present invention, the process of construction operations is reconstructed using a job process template with a specified output format to obtain the actual job process (i.e., the job process to be reviewed), making the job process more visual and facilitating the review of compliance. At the same time, different job process templates are used for different job scenarios, enabling the flexible acquisition of job processes under different job scenarios.

[0095] Regarding step 106, a compliance review is performed on the job process to be reviewed to obtain a review result, including:

[0096] Obtain the standard job process under the target job scenario;

[0097] Compare the standard job process with the job process to be reviewed to determine the review result including non-compliant process nodes and potential risk process nodes.

[0098] Specifically, compare the job process to be reviewed with the standard job process to identify deviations and non-compliances in the process steps, obtaining a list of differences. Among them, the compliance comparison includes: comparing the execution order, determining whether the execution order of the job process to be reviewed is consistent with that of the standard job process, and checking whether there are any missing standard steps in the actual job process to be reviewed, whether there are steps outside the standard process. Identify the steps with inconsistent execution order and the missing steps as non-compliant process nodes; and identify the steps outside the standard process as potential risk process nodes.

[0099] In a preferred embodiment, comparing the standard job process with the job process to be reviewed to determine the review result including non-compliant process nodes and potential risk process nodes includes:

[0100] Segment the job process to be reviewed to obtain a first type of job process corresponding to the standard job process and a second type of job process not corresponding to the standard job process;

[0101] Review the first type of job process according to the standard job process to determine non-compliant process nodes;

[0102] Determine each second type of job process as a potential risk process node;

[0103] Conduct a risk assessment on the non-compliant process nodes and potential risk process nodes to obtain a risk score, and determine a solution based on the risk score.

[0104] For example, as described in the previous example, the step number sequence of the operation process to be reviewed is 2, 1, 1, 3, 4, and the step number sequence of the standard operation process is 1, 2, 3, 4. Therefore, step 2 of the operation process to be reviewed is a non-compliant process node, and there is no potential risk process node. At this time, in the actual process, the construction team entered the construction area without completing the electrical equipment inspection, violating the safety production regulations, with potential safety hazards, which may lead to electrical accidents during the construction process. The reason is that the construction team did not strictly follow the process and lacked the confirmation of the electrical inspection status. The solution is to strengthen process training. The construction team can apply to enter the construction area only after the electrical inspection is completed and approved by the command center. It should be noted that for steps outside the standard process, the step number can be "0".

[0105] In the present invention, since not all operation processes in the actual operation process correspond to standard operation processes, for the second type of operation processes outside the standard process, they are all determined as potential risk process nodes. By performing risk assessment on non-compliant process nodes and potential risk process nodes, the risk score of the current operation process to be reviewed can be further determined, thereby further determining a suitable solution.

[0106] In a specific embodiment, the risk score is determined by the following formula:

[0107]

[0108] where M is the risk score; n1 is the number of non-compliant process nodes; α1, α2, α3 are the weight values corresponding to the operation personnel detection result, the precondition satisfaction result, and the compliance result respectively; p i , p ei are the actual number of people and the designated number of people during the operation of the i-th non-compliant process node respectively; m 2i is the precondition satisfaction result of the i-th non-compliant process node; m 3i is the compliance result of the i-th non-compliant process node; α1 + α2 + α3 = 1; n2 is the number of potential risk process nodes; m 4j is the risk score of the j-th potential risk process node; β is the influence factor of the operation process to be reviewed on the current operation, β > 0. Moreover, the higher the risk score value, the greater the risk.

[0109] It should be noted that the values of m 2i , m 3i are both 0 or 1, 0 < m 4j ≤ 1. Among them, when m 2i = 0, the precondition is satisfied; when m 2i = 1, the precondition is not satisfied. m 3iWhen it is 0, the compliance result meets the specification requirements; m 3i When it is 1, the compliance result does not meet the specification requirements. m 4j It is determined according to the empirical value. β is related to the impact of the operation process to be reviewed on the current operation. The greater the impact on the current operation, the higher the β value. If the current operation can run stably under the operation process to be reviewed, the β value is the smallest.

[0110] In this specific embodiment, the risk assessment of non-compliant process nodes includes whether the preconditions of the step where the non-compliant process node is located are met, whether the execution of the step where the non-compliant process node is located meets the specification requirements, and whether the number of operators meets the specified number of operators. For non-compliant process nodes, if all three results are not met, the higher the risk and the higher the potential safety hazard. It should be noted that the precondition is used to judge whether the precondition for the execution of the current non-compliant process node is met. Only when the precondition is met, the current non-compliant process node is more compliant.

[0111] In the present invention, the higher the risk score, the higher the risk level, and the higher and more urgent the potential safety hazard of the currently adopted operation process, and the more urgently the operation and maintenance personnel need to take corresponding solutions for safety inspection and adjustment.

[0112] In the present invention, by automatically collecting and processing all voice data during the operation process, the full coverage of the entire operation process is achieved, avoiding the omission problem caused by manual sampling inspection, and improving the review coverage rate. Utilizing the powerful semantic understanding ability of the large language model, the operation process information in the text data is accurately extracted, avoiding misjudgment caused by simple keyword matching, and improving the accuracy of compliance review. At the same time, an intelligent analysis is carried out using a pre-trained language model, reducing the dependence on a complex rule engine, reducing the complexity and cost of rule maintenance, and simplifying rule maintenance. Moreover, through the continuous learning and update of the large language model, the operation process compliance review method provided by the present invention can flexibly respond to changes in operation specifications in different industries and different scenarios, quickly adapt to new compliance requirements, and has higher adaptability. In addition, it can also determine violations, identify potential risks and hazards through compliance review, providing more comprehensive protection for operation safety and quality; and when the review result shows non-compliance, a clear explanation and solution are given, facilitating relevant personnel to understand and improve. The present invention significantly improves the efficiency and accuracy of the compliance review of operation voice conversations, providing strong support for the operation safety and quality management of industries such as logistics, mining, railway, and aviation.

[0113] Such as Figure 2 、 Figure 3 As shown, the embodiment of the present invention provides an operation process compliance review device based on voice recognition. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. At the hardware level, such asFigure 2 As shown in the figure, it is a hardware architecture diagram of a computing device where a workflow compliance review device based on speech recognition provided by an embodiment of the present invention. In addition to Figure 2 the shown processor, memory, network interface, and non-volatile memory, the computing device where the device is located in the embodiment usually may also include other hardware, such as a forwarding chip responsible for processing packets, etc. Taking software implementation as an example, as Figure 3 shown, as a logically meaningful device, it is formed by the CPU of its corresponding computing device reading the corresponding computer program in the non-volatile memory into the memory and running. A workflow compliance review device based on speech recognition provided by this embodiment, the device includes:

[0114] A speech recognition module 300, configured to recognize and convert the speech data collected in the target operation scenario to obtain target text data;

[0115] An analysis module 302, configured to input the target text data into a pre-trained language model, and output several groups of entities, events, the relationships between entities and entities and / or events, and the relationships between events; and according to the several groups of entities, events and relationships, obtain the workflow to be reviewed by using the target workflow template corresponding to the target operation scenario;

[0116] A review module 304, configured to perform compliance review on the workflow to be reviewed to obtain a review result.

[0117] In some specific implementation manners, the speech recognition module 300 may be used to execute the above step 100, the analysis module 302 may be used to execute the above step 102 and step 104, and the review module 304 may be used to execute the above step 106.

[0118] In an embodiment of the present invention, the entities include the department to which the staff belongs, the staff, the device name, the working area, the working geographical location, and the working time; the events include the actions of performing any operation task and the current status; the actions include completing inspection, waiting for a command, starting construction, and construction completion; the current status includes: all staff arrived, the device is being inspected, the device is ready, and the device has completed the operation; the relationships include the superior-subordinate relationships between the working departments to which the staff belong and the sequence of operations between operation tasks.

[0119] In an embodiment of the present invention, the speech recognition module 300 is further configured to perform the following operations:

[0120] Preprocess the speech data to obtain preprocessed speech data;

[0121] Mark the preprocessed speech data with a time stamp and location information;

[0122] Perform recognition and conversion on the preprocessed speech data to obtain temporary text data;

[0123] Correct and semantically error-correct the temporary text data according to the dialect knowledge base to obtain initial text data;

[0124] Sort the initial text data in chronological order of timestamps and correspondingly add position information to the initial text data to obtain target text data.

[0125] In one embodiment of the present invention, it further includes a training module, and the training module is used to perform the following operations:

[0126] Obtain the training parameters of the language model and freeze them; wherein, the language model is used to recognize the job full-process text data in the historical text data of the target job scenario; the job full-process text data includes several groups of entities, events, and association relationships;

[0127] Construct a low-rank matrix in the architecture of the pre-trained language model to obtain a fine-tuned language model;

[0128] Train the fine-tuned language model with at least two groups of first sample sets to obtain a pre-trained language model; wherein, each group of first sample sets includes the historical text data in the target job scenario as input and several groups of entities, events, and association relationships of the historical text data as output.

[0129] In one embodiment of the present invention, the training module is further used to perform the following operations:

[0130] Analyze the historical text data in the target job scenario to obtain the job full-process text data in the target job scenario;

[0131] The language model is trained through at least two groups of second sample sets; wherein, each group of second sample sets includes the historical text data as input and the job full-process text data of the historical text data as output.

[0132] In one embodiment of the present invention, the analysis module 302 is further used to perform the following operations:

[0133] Obtain the target job process template adopted by the target job scenario; wherein, different job scenarios adopt different job process templates; the target job process template is used to output target entities, target events, and target association relationships in a specified output format;

[0134] Determine several groups of target entities, target events, and target association relationships from several groups of entities, events, and association relationships according to the specified output format;

[0135] Output several groups of target entities, target events, and target association relationships in a specified output format and chronological order to obtain a job process to be reviewed.

[0136] In an embodiment of the present invention, the review module 304 is further configured to perform the following operations:

[0137] Obtain the standard job process in the target job scenario;

[0138] Segment the job process to be reviewed to obtain a first type of job process corresponding to the standard job process and a second type of job process not corresponding to the standard job process;

[0139] Review the first type of job process according to the standard job process to determine non-compliant process nodes;

[0140] Determine each second type of job process as a potential risk process node;

[0141] Conduct a risk assessment on the non-compliant process nodes and potential risk process nodes to obtain a risk score, and determine a solution based on the risk score.

[0142] It can be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on a job process compliance review device based on speech recognition. In other embodiments of the present invention, a job process compliance review device based on speech recognition may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0143] For the information interaction, execution process, etc. between the modules in the above device, since they are based on the same concept as the method embodiments of the present invention, the specific content can be referred to the description in the method embodiments of the present invention and will not be elaborated here.

[0144] The embodiments of the present invention also provide a computing device, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements a job process compliance review method based on speech recognition in any embodiment of the present invention.

[0145] The embodiments of the present invention also provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, it causes the processor to execute a job process compliance review method based on speech recognition in any embodiment of the present invention.

[0146] Embodiments of the present application also provide a computer program product. The computer program product includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes a method for compliance review of an operation process based on speech recognition described in any one of the above embodiments.

[0147] Specifically, a system or device equipped with a storage medium can be provided. Software program code for implementing the functions of any one of the above embodiments is stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0148] In this case, the program code read from the storage medium itself can implement the functions of any one of the above embodiments. Therefore, the program code and the storage medium storing the program code constitute a part of the present invention.

[0149] Embodiments of the storage medium for providing the program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer via a communication network.

[0150] In addition, it should be clear that not only can the functions of any one of the above embodiments be implemented by executing the program code read by the computer, but also by instructing the operating system on the computer based on the program code to complete part or all of the actual operations.

[0151] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion module connected to the computer. Subsequently, based on the instructions of the program code, the CPU etc. installed on the expansion board or expansion module execute part and all of the actual operations, thereby implementing the functions of any one of the above embodiments.

[0152] It should be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

[0153] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including those of the above method embodiments; and the aforementioned storage medium includes various media such as ROM, RAM, magnetic disks or optical discs that can store program codes.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for compliance review of operation processes based on speech recognition, characterized in that, Including: Performing recognition and conversion on the voice data collected in the target operation scenario to obtain target text data; Inputting the target text data into a pre-trained language model to output several groups of entities, events, the relationships between entities and entities and / or events, and the relationships between events; the entities include the department to which the staff belongs, the staff, the equipment name, the work area, the work geographical location, and the working time; the events include the actions and current status of performing any operation task; the relationships include the hierarchical relationship between the departments to which the staff belong and the sequence of operation tasks; According to several groups of the entities, the events, and the relationships, using the target operation process template corresponding to the target operation scenario to obtain the operation process to be reviewed; Obtaining the standard operation process under the target operation scenario; Comparing the standard operation process and the operation process to be reviewed to determine the review result including non-compliant process nodes and potential risk process nodes, and the risk score is determined by the following formula: Among them, M is the risk score; n1 is the number of non-compliant process nodes; α1, α2, and α3 are the weight values corresponding to the operation personnel detection result, the precondition satisfaction result, and the compliance result respectively; p i , p ei are the actual number of people and the designated number of people during the operation of the i-th non-compliant process node respectively; m 2i is the precondition satisfaction result of the i-th non-compliant process node; m 3i is the compliance result of the i-th non-compliant process node; n2 is the number of potential risk process nodes; m 4j is the risk score of the j-th potential risk process node; β is the influence factor of the operation process to be reviewed on the current operation.

2. The method according to claim 1, wherein The performing recognition and conversion on the voice data collected in the target operation scenario to obtain target text data includes: Performing preprocessing on the voice data to obtain preprocessed voice data; Marking time stamps and location information on the preprocessed voice data; Performing recognition and conversion on the preprocessed voice data to obtain temporary text data; Correcting and semantically correcting the temporary text data according to the dialect knowledge base to obtain initial text data; Sorting the initial text data in the chronological order of the time stamps and correspondingly adding the location information to the initial text data to obtain the target text data.

3. The method according to claim 1, wherein The pre-trained language model is trained by the following method: Obtaining and freezing the training parameters of the language model; wherein, the language model is used to identify the operation full-process text data in the historical text data of the target operation scenario; the operation full-process text data includes several groups of entities, events, and relationships; Constructing a low-rank matrix in the architecture of the pre-trained language model to obtain a fine-tuned language model; Training the fine-tuned language model with at least two groups of first sample sets to obtain the pre-trained language model; wherein, each group of the first sample sets includes the historical text data in the target operation scenario as the input and several groups of entities, events, and relationships of the historical text data as the output.

4. The method according to claim 3, characterized in that The language model is trained by the following method: Analyzing the historical text data in the target operation scenario to obtain the operation full-process text data in the target operation scenario; The language model is trained by at least two groups of second sample sets; wherein, each group of the second sample sets includes the historical text data as the input and the operation full-process text data of the historical text data as the output.

5. The method according to claim 1, wherein The actions include completing inspection, waiting for commands, starting construction, and construction completed; the current status includes: all staff arrived, equipment being inspected, equipment ready, and equipment completed the operation; and / or Said obtaining a to-be-reviewed operation process according to a plurality of groups of the entities, the events, and the association relationships by using a target operation process template corresponding to the target operation scenario includes: Obtaining the target operation process template adopted by the target operation scenario; wherein, operation process templates adopted by different operation scenarios are different; the target operation process template is used to output target entities, target events, and target association relationships in a specified output format; Determining a plurality of groups of the target entities, the target events, and the target association relationships from the plurality of groups of the entities, the events, and the association relationships according to the specified output format; Outputting the plurality of groups of the target entities, the target events, and the target association relationships in the specified output format and chronological order to obtain the to-be-reviewed operation process.

6. The method according to claim 1, wherein Said comparing the standard operation process and the to-be-reviewed operation process to determine a review result including non-compliant process nodes and potential risk process nodes includes: Segmenting the to-be-reviewed operation process to obtain a first type of operation process corresponding to the standard operation process and a second type of operation process not corresponding to the standard operation process; Reviewing the first type of operation process according to the standard operation process to determine non-compliant process nodes; Determining each of the second type of operation processes as potential risk process nodes; Performing a risk assessment on the non-compliant process nodes and the potential risk process nodes to obtain a risk score, and determining a solution based on the risk score.

7. An operation process compliance review device based on speech recognition, which is used to implement the method described in any one of claims 1 to 6, and is characterized in that Including: A voice recognition module, configured to recognize and convert voice data collected in a target operation scenario to obtain target text data; An analysis module, configured to input the target text data into a pre-trained language model, output a plurality of groups of entities, events, association relationships between entities and / or between events, and between events; and obtain a to-be-reviewed operation process according to a plurality of groups of the entities, the events, and the association relationships by using a target operation process template corresponding to the target operation scenario; A review module, configured to perform a compliance review on the to-be-reviewed operation process to obtain a review result.

8. A computing device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1-6 is implemented.

9. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1-6.

Images