Process operation step inspection method and device, electronic equipment and readable storage medium

Abstract

The application discloses a process operation step inspection method and device, electronic equipment and a readable storage medium, and is applied to the technical field of machine vision. Wherein, the method comprises: in response to a mapping relationship establishment instruction, establishing a state-graph corresponding relationship for the image recognition result of a plurality of process operation images at the same time and the corresponding state of the state machine; determining the state transition event uniquely corresponding to each state according to the state-graph corresponding relationship; inputting the process operation image acquired at the current time into the pre-trained target recognition model to obtain a target recognition result, and inputting the target recognition result into the state machine, and the state machine determines to enter the corresponding state according to the target recognition result and each state transition event; and according to the state migration change information of the state machine, whether the process operation step conforms to the standard is verified, so that the technical defects that the related technology cannot be associated with multiple targets, cannot be quickly customized and is inconvenient for experience accumulation can be solved.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a method, apparatus, electronic device and readable storage medium for inspecting process operation steps. Background Technology

[0002] In the production process, the series of activities undertaken to transform raw materials into downstream finished products are called processes. The various operational steps within a process must be rational and orderly to ensure the stability and reliability of the finished product quality. Therefore, it is necessary to inspect the process operation steps. Process operation step inspection refers to checking whether the operation steps during production are standardized. When non-standard operations occur, timely warnings, alarms, or direct intervention should be issued to prevent accidents.

[0003] Traditional process inspection relies on manual supervision, which can be categorized from several dimensions: From a management perspective, process inspection steps are divided into traditional process verification (e.g., first-piece verification, process verification) and lifecycle-based verification (e.g., process design, process validation, process continuous validation); from an equipment perspective, these include human eyes, video equipment, and image equipment (conventional equipment); from a legal perspective, they include manual monitoring, establishing inspection procedures, and finished product sampling (primarily for inspecting immediate results); and from a technical perspective, they include discrete-time verification rules, visual judgment, and signal transmission (conventional inspection techniques). However, this manual process inspection technique primarily targets inspections before and after the process operation, lacking methods for automated inspection of the process itself. It is easily influenced by human subjectivity, inevitably leading to errors, and makes sharing and promoting solutions difficult.

[0004] With the rapid development of technologies such as artificial intelligence and knowledge engineering, the branches of artificial intelligence are becoming increasingly diverse, and machine vision is one such branch. Machine vision utilizes machines to perform measurement tasks and judgment tasks that require human eyes. Currently, it has made breakthroughs in areas such as automatic acquisition of key information, knowledge representation and reasoning learning, and graph-based deep learning, and has been widely applied in identification, measurement, detection, and positioning technologies across numerous industries, including electronics, industrial control, new energy, semiconductors, and medicine. For example, in the new energy industry, machine vision can identify the polarity of battery cells, measure battery dimensions, detect defects in solder joints, and locate battery pack positions. Machine vision systems use machine vision products (such as image acquisition devices) to convert the target into an image signal, which is then transmitted to a dedicated image processing system. Once the image processing system acquires the image information, it analyzes it based on pixel distribution, brightness, color, and other image information, converting the analysis results back into digital signals. The image system then classifies, analyzes, and processes these digital signals, making judgments based on the target's characteristics, and using the judgment results to control various devices.

[0005] Compared to human vision, machine vision has significant advantages in terms of light sensitivity range, speed, and environmental requirements. It is particularly useful in hazardous working environments, dangerous conditions unsuitable for manual labor, or tedious tasks requiring substantial human and financial resources. Therefore, to overcome the shortcomings of traditional methods, related technologies typically employ single-step verification methods based on machine vision, such as... Figure 1 and Figure 2 As shown, this method involves multi-point annotation of images, target detection, and determining the correctness of a single process step based on whether a specific target is detected. However, this method often involves customized process step verification or process verification methods tailored to specific scenarios in specific industries. The training model is customized for a single scenario and single step, resulting in low overall efficiency and a lack of universal applicability. Furthermore, the verification logic is independent, lacking overall coherence and predictable sequence, making it unsuitable for complex verification scenarios with multiple checkpoints that are sequential and interconnected. Moreover, the related technologies employ process operation step verification methods based on single neural network training models or multi-neural network training models based on pipelines, which hinders professionals from transferring and solidifying their experience, leading to a poor user experience. Summary of the Invention

[0006] This application provides a method, apparatus, electronic device, and readable storage medium for inspecting process operation steps, which solves the technical drawbacks of related technologies, such as the inability to correlate multiple targets, the inability to quickly customize, and the inconvenience of accumulating experience.

[0007] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions:

[0008] One embodiment of the present invention provides a method for inspecting process operation steps, including:

[0009] In response to the mapping relationship establishment instruction, a state-graph correspondence is established between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine;

[0010] Based on the state-graph correspondence, a unique state transition event is determined for each state;

[0011] The process operation image acquired at the current moment is input into the pre-trained target recognition model to obtain the target recognition result, and the target recognition result is input into the state machine. The state machine determines the corresponding state to enter based on the target recognition result and each state transition event.

[0012] Based on the state transition information of the state machine, verify whether the process operation steps meet the standards.

[0013] Optionally, the response mapping relationship establishment instruction establishes a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine, including:

[0014] In response to the region segmentation command, the corresponding ROI region is matched for each process operation image;

[0015] In response to the mapping relationship establishment instruction, a state-graph correspondence is established between the identification results of each ROI region at the same time and the corresponding state of the state machine.

[0016] Optionally, the state-graph correspondence further includes the state transition sequence, which corresponds to the target action trajectory; the step of verifying whether the process operation steps conform to the standard based on the state transition information of the state machine includes:

[0017] Determine whether the state transition information is consistent with the state transition sequence;

[0018] If so, then the process operation steps meet the standards.

[0019] Optionally, after establishing a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine, the response mapping relationship establishment instruction further includes:

[0020] If no state change of the state machine is detected within the target time window, an operation timeout message is generated.

[0021] Optionally, before establishing the state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine, the response mapping relationship establishment instruction further includes:

[0022] For a target process operation that meets preset conditions, acquire process operation images corresponding to the target process operation from multiple image acquisition devices at the same time from different angles.

[0023] Based on the target recognition results of each process operation image of the target process operation, the image recognition result corresponding to the target process operation is generated.

[0024] Optionally, the response mapping relationship establishment instruction establishes a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine, including:

[0025] In response to the mapping relationship establishment command, the state machine editing tool is invoked in the visual configuration mode to establish a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine.

[0026] Optionally, after verifying whether the process operation steps conform to the standard based on the state transition information of the state machine, the method further includes:

[0027] If the process operation steps are determined to be non-compliant with the standard based on the state machine transition information, an alarm message will be generated.

[0028] Another embodiment of the present invention provides a process operation step inspection device, comprising:

[0029] The mapping relationship binding module is used to respond to the mapping relationship establishment command and establish a state-graph mapping relationship between the image recognition results of multiple frames of process operation images at the same time and the corresponding state of the state machine.

[0030] The event determination module is used to determine the unique state transition event corresponding to each state based on the state-graph correspondence.

[0031] The state machine detection module is used to input the process operation image acquired at the current moment into a pre-trained target recognition model to obtain the target recognition result, and input the target recognition result into the state machine. The state machine determines the corresponding state to enter based on the target recognition result and each state transition event.

[0032] The step detection module is used to verify whether the process operation steps conform to the standard based on the state transition information of the state machine.

[0033] This invention also provides an electronic device, including a processor, which executes a computer program stored in a memory to implement the steps of the process operation step inspection method as described in any of the preceding claims.

[0034] Finally, this embodiment of the invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the process operation step inspection method as described in any of the preceding claims.

[0035] The advantages of the technical solution provided in this application lie in its seamless integration of machine vision target detection with a state machine. By using the state machine to organically combine the detected targets, it solves the technical drawbacks of machine vision detection results being discrete and unable to be mutually verified, requiring extensive manual confirmation. Based on the customizable nature of the state machine, the inspection logic for the entire process operation step can be rapidly customized, effectively improving overall efficiency and enabling multi-target association, thus offering better universality. Furthermore, by integrating process operation step recognition and state transition judgment, it allows professionals to directly participate and solidify their process experience, effectively improving the user experience. Therefore, this application, by combining machine vision technology with state machine operation management of process steps, achieves automated analysis and inspection of process step specifications, making it more practical.

[0036] Furthermore, embodiments of the present invention also provide corresponding implementation devices, electronic devices, and readable storage media for the process operation step inspection method, further making the method more practical, and the devices, electronic devices, and readable storage media have corresponding advantages.

[0037] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a machine vision process inspection method provided in the embodiments of the present invention.

[0040] Figure 2 A schematic diagram illustrating an exemplary application scenario of the related technologies provided in the embodiments of the present invention;

[0041] Figure 3A flowchart illustrating a method for inspecting process operation steps provided in an embodiment of the present invention;

[0042] Figure 4 A flowchart illustrating another process operation step inspection method provided in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram illustrating the delineation of the Region of Interest (ROI) in an illustrative example provided for an embodiment of the present invention;

[0044] Figure 6 A schematic diagram illustrating the establishment of correspondences in an illustrative example provided for an embodiment of the present invention;

[0045] Figure 7 A schematic diagram illustrating the matching of operation steps with state machine states in an illustrative example provided for an embodiment of the present invention;

[0046] Figure 8 A schematic diagram of state transition changes provided in an illustrative example of an embodiment of the present invention;

[0047] Figure 9 A structural diagram of a specific embodiment of the process operation step inspection device provided in this invention;

[0048] Figure 10 This is a structural diagram of a specific embodiment of the electronic device provided in this invention. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The terms "first," "second," "third," "fourth," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units not listed.

[0051] After introducing the technical solutions of the embodiments of the present invention, the various non-limiting embodiments of this application will be described in detail below.

[0052] First see Figure 3 , Figure 3 This is a flowchart illustrating a method for inspecting process operation steps according to an embodiment of the present invention. The embodiment of the present invention may include the following:

[0053] S301: Response to the mapping relationship establishment instruction, establish a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine.

[0054] In this embodiment, the mapping relationship establishment command is issued by a user, such as a professional, through a human-computer interaction interface. The professional, combining their experience, establishes mapping relationships based on the various operational steps of the process in the current application scenario. This mapping relationship transforms and solidifies their experience, making it easily applicable to various processes and discrete industries with related operational step verification or target trajectory requirements. The state machine operating environment maintains the process operation step identification state and performs process operation step checks sequentially. Image recognition results are obtained through a machine vision model. The machine vision model inference environment runs a pre-trained model, where the process operation steps that the model can recognize have been pre-trained and optimized, maximizing the reuse of the pre-trained model results. Based on the customizable characteristics of the state machine, the entire process operation step verification logic can be quickly customized, achieving multi-step, i.e., multi-target association. The state machine organically combines the detection targets, thereby achieving automated detection, effectively reducing manual labor, and solving the problem in related technologies where machine vision detection results are discrete and cannot be mutually verified, requiring extensive manual confirmation. Image recognition results include the recognition results of the target of interest in each frame of the process operation image at the same time. Each frame of the industrial operation image is an image of each process step implemented in the current process; that is, the number of industrial steps required for the current process is the same as the number of frames in the process operation image. The total number of states contained in the state machine is the same as the total number of process operation steps. From completing one process step to executing the next process step, it corresponds to a jump from one state to another corresponding state. The state-graph correspondence is the image recognition result of the process operation image corresponding to each process operation in a specified time window for each state of the state machine. In other words, it is the standard operation of each process operation in each state of the state machine, which is identified by the image recognition result of the process operation image collected for the corresponding process operation.

[0055] S302: Based on the state-diagram correspondence, determine the unique state transition event corresponding to each state.

[0056] After establishing the state-graph correspondence in the previous step, a state transition event can be set for each state machine to trigger state transitions. This state transition event can be identified using information that uniquely identifies the current state or the current process operation step. For example, in a production line process, only one process operation occurs at a time. Therefore, only the image recognition result of one of the process operation images at that time can uniquely identify the current process operation step or state. This image recognition result can then serve as the state transition event.

[0057] S303: Input the process operation image acquired at the current moment into the pre-trained target recognition model to obtain the target recognition result, and input the target recognition result into the state machine. The state machine determines the corresponding state to enter based on the target recognition result and each state transition event.

[0058] In this embodiment, the target recognition model can employ any machine learning model, such as an artificial neural network or a YOLOv5 network. A network model with good convergence is obtained by training a large number of image samples. The target recognition model is used to perform image recognition on the input process operation image. The target recognition result is the sum of the image recognition results of the target of interest in all process operation images at the current moment. This target recognition result is input into the state machine. Since the state transition event has been determined in step S302, comparing the state transition event with the target recognition result determines whether the input process operation image corresponds to any state of the state machine. If they correspond, the state machine is triggered to jump to the corresponding state.

[0059] S304: Based on the state transition information of the state machine, verify whether the process operation steps meet the standards.

[0060] In this embodiment, the state transition information refers to the changes in each state of the state machine from startup to entering the final state. This includes, for example, which states the state machine has passed through, the order of these changes, and the duration of each state. Since states correspond to process operation steps, state transitions can reflect whether the process operation steps meet preset operation standards. If the state transition information determines that the process operation steps do not meet the standards, an alarm message is generated, such as a voice announcement of an operation violation. Figure 4 As shown.

[0061] In the technical solution provided by this invention, machine vision target detection is seamlessly integrated with a state machine. The state machine organically combines the detected targets, thereby solving the technical drawbacks of machine vision detection results being discrete and unable to be mutually verified, requiring extensive manual confirmation. Based on the customizable nature of the state machine, the inspection logic of the entire process operation step can be rapidly customized, which not only effectively improves overall efficiency but also enables multi-target association, resulting in better universality. By integrating process operation step recognition and state transition judgment, professionals can directly participate and solidify their process experience, effectively improving the user experience. Therefore, this application, by combining machine vision technology with state machine operation management of process steps, achieves automated analysis and inspection of process step specifications, making it more practical.

[0062] To further improve the efficiency and accuracy of target recognition result generation, and considering that processing the entire region during image inference in the target recognition model training process would lead to low overall efficiency, based on the above embodiments, this application can pre-define the region of interest (ROI) for each process operation image, and only image recognition needs to be performed on the ROI region. Accordingly, an optional implementation of the above S301 step is as follows:

[0063] In response to the region segmentation command, the corresponding ROI region is matched for each process operation image;

[0064] In response to the mapping relationship establishment instruction, a state-graph correspondence is established between the identification results of each ROI region at the same time and the corresponding state of the state machine.

[0065] To further improve practicality, this embodiment can also determine the execution order among multiple targets. Correspondingly, the state-graph correspondence can also include the state transition sequence, which corresponds to the target's trajectory. Based on the above embodiment, it can also include:

[0066] Determine whether the state transition information is consistent with the state transition sequence;

[0067] If so, then the process operation steps meet the standards.

[0068] In this embodiment, for process scenarios with specific target trajectory requirements, the target can be identified and a target trajectory route can be defined. The change order of each state in the state transition information can be used to determine whether the target movement conforms to the trajectory route, thereby determining whether the target is operating according to the preset trajectory route. If the target does not operate according to the preset trajectory route, it can be determined as a violation and an alarm can be triggered.

[0069] To further improve practicality, this embodiment can also identify whether delays occur during the process operation. Based on the above embodiment, it may include the following:

[0070] As an optional implementation, during the process operation, a timeout can be determined by checking the state at a specified moment, i.e., whether the process is stuck at a certain process step. Optionally, if no state change of the state machine is detected within the target time window, an operation timeout message is generated.

[0071] As another optional implementation, the entire process operation can be evaluated by setting a standard time. Optionally, a response time parameter is assigned as an instruction to set a standard duration for each state of the state machine; the duration of each state is determined based on the state transition information of the state machine; if the duration of each state does not exceed the corresponding standard duration, then the process operation steps meet the standard.

[0072] To further improve the accuracy of process operation inspection, based on the above embodiments, it may also include:

[0073] For a target process operation that meets preset conditions, acquire process operation images corresponding to the target process operation from multiple image acquisition devices at the same time from different angles.

[0074] Based on the target recognition results of each process operation image of the target process operation, generate the image recognition results corresponding to the target process operation.

[0075] The preset conditions are factors that affect the accuracy of image recognition. For example, there may be situations where the target is too small to be recognized. The same target from multiple angles of multiple cameras can be normalized before recognition to improve the accuracy of target recognition.

[0076] As can be seen from the above, for application scenarios where the target recognition accuracy of a single monitoring screen is low, this embodiment can normalize the same target in multiple monitoring screens and improve usability by combining it with a state machine.

[0077] To enhance the user experience, the state machine editing tool employs a user-friendly visual editing interface, allowing users to edit basic information based on the characteristics of the finite state machine, such as adding state nodes, adding connections, dragging, and deleting. Based on the above embodiments, it may also include:

[0078] In response to the mapping relationship establishment command, the state machine editing tool is invoked in the visual configuration mode to establish a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine.

[0079] The state machine editing tool in this embodiment adopts a visual configuration method, which is separated from the state machine operation. It is easy to operate, gives full play to the flexibility of the state machine, and covers application scenarios.

[0080] The above embodiments, by introducing machine vision technology, pre-train target recognition models for operation steps in different industries, which can automatically identify whether operation steps in the production process meet process standard requirements. Combined with a customizable key step status management mechanism, it further judges whether the order of operation steps is reasonable and whether the time is up to standard. When an operation step that does not meet the expected state occurs, it provides prompts or issues alarms, thereby supporting the inspection of multiple related complex steps. For certain specific operation steps, a direction-based detection method can be used to determine whether the process step conforms to a predetermined trajectory, thereby performing more refined process operation step inspection. To enable those skilled in the art to more clearly understand the technical solution of this application, this application also provides an illustrative embodiment, which may include the following:

[0081] For large-scale landfill applications, when drivers operate garbage trucks to dump garbage into the landfill pit, specific safety procedures must be followed:

[0082] After the garbage truck comes to a complete stop, the driver must leave the cab (S1: no one is in the cab) before proceeding to the next step.

[0083] There is a safety rope in front of the vehicle. The safety rope must be pulled out and secured to the pull-out stake (S2: the pull-out stake is already secured) to prevent the vehicle from sliding into the landfill pit.

[0084] At this point, the driver needs to move away from the front and rear of the vehicle and walk to a safe area on the side of the vehicle (S3: Driver is in position) before operating the tipper to dump the garbage.

[0085] To make this method more universally applicable, more states (S4: Other) can be set for other regions according to actual needs.

[0086] To improve processing efficiency, the above steps correspond to different areas (i.e., ROI areas) of the same monitoring screen, such as... Figure 5 As shown. Based on this application scenario, the detection process for the process operation steps is as follows:

[0087] In response to the mapping relationship establishment instruction, a state-graph correspondence is established between the image recognition results of each ROI region at the same time and the corresponding state of the state machine. This means binding the target recognition model and the state machine together. Figure 6 As shown.

[0088] Based on the state-diagram correspondence, a unique state transition event is determined for each state. This embodiment corresponds to four steps, including four states: S1, S2, S3, and S4. The state transition events for each state are Event1 (occupant in the cab), Event2 (anti-pull-out stake secured), Event3 (occupant in the safe zone), and Event4 (other and more states), such as... Figure 7 As shown, when a state transition event is detected, the system can jump to the corresponding state.

[0089] The process operation image acquired at the current moment is input into a pre-trained target recognition model to obtain the target recognition result. This result is then input into a state machine, which determines the appropriate state to enter based on the target recognition result and various state transition events. The execution process of the state machine includes:

[0090] Step 1: Before starting the operation step detection, the state machine is in the START initial state;

[0091] Step 2: After Event1 is detected, enter state S1 and record S1; other non-Event1 events directly enter the corresponding state.

[0092] Step 3: After detecting Event2 based on S1, enter state S2 and record S2; other non-Event2 events enter their corresponding states.

[0093] Step 4: After detecting Event3 based on S2, enter state S3 and record S3; other non-Event3 events enter their corresponding states.

[0094] Step 5: After detecting Event4 based on S3, enter state S4 and record S4; other non-Event4 events enter their corresponding states.

[0095] After the above steps, the following is obtained: Figure 8 The state transition information shown determines whether the state changes occur in the order of S1 => S2 => S3 => S4 after entering the final state (END) or after the time period expires. If not, the state is considered an operation violation. If the state machine does not change within the specified time window, the operation is considered untimely.

[0096] After completing the current round of inspection, the state machine can be reset to its initial state (i.e., START) to begin a new round of inspection.

[0097] This application's technical solution uses machine vision-based process operation step recognition, combined with a process operation step matching state machine, to verify whether the operation steps are standardized and timely under different process requirements. This enables automated recognition of personnel or machine process operation steps in both process and discrete industries, providing user-friendly prompts or system alarms. Furthermore, the editable and customizable state machine normalizes the verification rules for different personnel or machine process operation steps across various industries, thereby quickly translating the experience of professionals into practice.

[0098] It should be noted that there is no strict order of execution between the steps in this application. As long as they conform to the logical order, these steps can be executed simultaneously or in a certain preset order. The above figures are only illustrative and do not represent that the execution order can only be like this.

[0099] This invention also provides a corresponding apparatus for the process operation step inspection method, further enhancing the practicality of the method. The apparatus can be described from both a functional module perspective and a hardware perspective. The process operation step inspection apparatus provided in this invention is described below, and the process operation step inspection apparatus described below can be referred to in correspondence with the process operation step inspection method described above.

[0100] From the perspective of functional modules, see Figure 9 , Figure 9 A structural diagram of a process operation step inspection device provided in an embodiment of the present invention is shown in one specific implementation. The device may include:

[0101] The mapping relationship binding module 901 is used to respond to the mapping relationship establishment instruction and establish a state-graph mapping relationship between the image recognition results of multiple frames of process operation images at the same time and the corresponding state of the state machine.

[0102] The event determination module 902 is used to determine the unique state transition event corresponding to each state based on the state-diagram correspondence.

[0103] The state machine detection module 903 is used to input the process operation image acquired at the current moment into the pre-trained target recognition model to obtain the target recognition result, and input the target recognition result into the state machine. The state machine determines the corresponding state to enter based on the target recognition result and each state transition event.

[0104] The step detection module 904 is used to check whether the process operation steps conform to the standard based on the state transition information of the state machine.

[0105] Optionally, in some embodiments of this example, the above-mentioned correspondence binding module 901 may be further used to: respond to the region division instruction, match the corresponding ROI region for each process operation image; respond to the mapping relationship establishment instruction, construct a state-graph correspondence between the identification results of each ROI region at the same time and the corresponding state of the state machine.

[0106] As an optional implementation of this embodiment, the above-mentioned step detection module 904 can also be used to: the state-graph correspondence relationship also includes the state transition change sequence, which corresponds to the target movement trajectory route. It determines whether the state transition change information is consistent with the state transition change sequence; if so, the process operation steps meet the standard.

[0107] As another optional implementation of this embodiment, the above-mentioned device may further include a delay detection module, which generates operation timeout information if no state change of the state machine is detected within the target time window.

[0108] As another optional implementation of this embodiment, the above-mentioned device may further include an image processing module, which is used to acquire process operation images corresponding to the target process operation acquired by multiple image acquisition devices from different angles at the same time for the target process operation that meets preset conditions; and to generate image recognition results corresponding to the target process operation based on the target recognition results of each process operation image of the target process operation.

[0109] Optionally, in some other embodiments of this example, the above-mentioned correspondence binding module 901 may be further used to: respond to the mapping relationship establishment instruction, call the state machine editing tool in the visual configuration mode to establish a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine.

[0110] Optionally, in some other embodiments of this example, the above-mentioned device may also include an alarm module, which generates an alarm message if it is determined from the state machine transition information that the process operation steps do not conform to the standard.

[0111] The functions of each functional module of the process operation step inspection device described in the embodiments of the present invention can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0112] As can be seen from the above, the embodiments of the present invention solve the technical drawbacks of related technologies, such as the inability to associate multiple targets, the inability to quickly customize, and the inconvenience of accumulating experience.

[0113] The process operation step inspection device mentioned above is described from the perspective of functional modules. Furthermore, this application also provides an electronic device, which is described from the perspective of hardware. Figure 10 This is a schematic diagram of the structure of the electronic device provided in one embodiment of this application. For example... Figure 10 As shown, the electronic device includes a memory 100 for storing a computer program; and a processor 101 for executing the computer program to implement the steps of the process operation step inspection method as described in any of the above embodiments.

[0114] The processor 101 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 101 may also be a controller, microcontroller, microprocessor, or other data processing chip. The processor 101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0115] The memory 100 may include one or more computer-readable storage media, which may be non-transitory. The memory 100 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the memory 100 may be an internal storage unit of an electronic device, such as a server hard drive. In other embodiments, the memory 100 may be an external storage device of an electronic device, such as a plug-in hard drive on a server, a Smart Media Card (SMC), a Secure Digital (SD) card, or a Flash Card. Furthermore, the memory 100 may include both internal and external storage units of the electronic device. The memory 100 can be used not only to store application software and various types of data installed on the electronic device, such as code in the process of executing the process operation step inspection method, but also to temporarily store data that has been output or will be output. In this embodiment, the memory 100 is used to store at least the following computer program 1001, which, after being loaded and executed by the processor 101, is capable of implementing the relevant steps of the process operation step inspection method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 100 may also include an operating system 1002 and data 1003, and the storage method may be temporary storage or permanent storage. The operating system 1002 may include Windows, Unix, Linux, etc. The data 1003 may include, but is not limited to, data corresponding to the inspection results of process operation steps.

[0116] In some embodiments, the aforementioned electronic device may further include a display screen 102, an input / output interface 103, a communication interface 104 (or network interface), a power supply 105, and a communication bus 106. The display screen 102 and input / output interface 103, such as a keyboard, are user interfaces; optional user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a display screen or display unit, used to display information processed in the electronic device and to display a visual user interface. The communication interface 104 may optionally include a wired interface and / or a wireless interface, such as a Wi-Fi interface, a Bluetooth interface, etc., typically used to establish communication connections between the electronic device and other electronic devices. The communication bus 106 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0117] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, such as sensors 107 that perform various functions.

[0118] The functions of each functional module of the electronic device described in the embodiments of the present invention can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0119] As can be seen from the above, the embodiments of the present invention solve the technical drawbacks of related technologies, such as the inability to associate multiple targets, the inability to quickly customize, and the inconvenience of accumulating experience.

[0120] It is understood that if the process operation step inspection method in the above embodiments is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, and other media capable of storing program code.

[0121] Based on this, embodiments of the present invention also provide a readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the process operation step verification method described in any of the above embodiments are as follows.

[0122] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the hardware disclosed in the embodiments, including devices and electronic equipment, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0123] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0124] The foregoing has provided a detailed description of a process operation step inspection method, apparatus, electronic device, and readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the invention. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for inspecting process operation steps, characterized in that, include: For different industry process operation steps that correspond to different areas of the same monitoring screen, respond to the area division command and match the corresponding ROI area for each process operation image; In response to the mapping relationship establishment command, the state machine editing tool is invoked in a visual configuration mode to construct a state-graph correspondence between the identification results of each ROI region at the same time and the corresponding state of the state machine. The standard operation of each process operation in each state of the state machine is identified by the image recognition results of the process operation images collected for the corresponding process operation. The total number of states contained in the state machine is the same as the total number of process operation steps. The state-graph correspondence also includes the state transition sequence, which corresponds to the target action trajectory route. Based on the state-graph correspondence, a unique state transition event is determined for each state; The process operation image acquired at the current moment is input into the pre-trained target recognition model to obtain the target recognition result. The target recognition result is then input into the state machine. The state machine determines the corresponding state to enter based on the target recognition result and each state transition event: when the first state transition event is detected, the corresponding first state is entered and recorded. Other non-first state transition events directly enter the corresponding state. When the second state transition event is detected based on the first state, the second state is entered and recorded. Other non-second state transition events enter the corresponding state. Based on the state transition information of the state machine, the process operation steps are checked to see if they meet the standards: when entering the final state or after the time period is reached, it is determined whether the state change occurs in the order of state transition. If not, the state is determined to be an operation violation. If the state machine state does not change within the specified time window, the operation is determined to be untimely. The state transition information refers to the changes of each state of the state machine from startup to entering the final state.

2. The method for inspecting process operation steps according to claim 1, characterized in that, The response mapping relationship establishment instruction, after establishing a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine, also includes: If no state change of the state machine is detected within the target time window, an operation timeout message is generated.

3. The method for inspecting process operation steps according to claim 1, characterized in that, Before establishing the state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine, the response mapping relationship establishment instruction also includes: For a target process operation that meets preset conditions, acquire process operation images corresponding to the target process operation from multiple image acquisition devices at the same time from different angles. Based on the target recognition results of each process operation image of the target process operation, the image recognition result corresponding to the target process operation is generated.

4. The method for inspecting process operation steps according to any one of claims 1 to 3, characterized in that, The response mapping relationship establishment instruction establishes a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine, including: In response to the mapping relationship establishment command, the state machine editing tool is invoked in the visual configuration mode to establish a state-graph correspondence between the image recognition results of multiple frames of process operation images at the same time and the corresponding states of the state machine.

5. The method for inspecting process operation steps according to claim 4, characterized in that, After verifying whether the process operation steps conform to the standard based on the state transition information of the state machine, the process further includes: If the process operation steps are determined to be non-compliant with the standard based on the state machine transition information, an alarm message will be generated.

6. A process operation step inspection device, characterized in that, include: The mapping relationship binding module is used to assign different regions of the same monitoring screen to different process operation steps in different industries. In response to the region division command, it matches the corresponding ROI region for each process operation image. In response to the mapping relationship establishment command, the state machine editing tool is invoked in a visual configuration mode to construct a state-graph correspondence between the identification results of each ROI region at the same time and the corresponding state of the state machine. The standard operation of each process operation in each state of the state machine is identified by the image recognition results of the process operation images collected for the corresponding process operation. The total number of states contained in the state machine is the same as the total number of process operation steps. The state-graph correspondence also includes the state transition sequence, which corresponds to the target action trajectory route. The event determination module is used to determine the unique state transition event corresponding to each state based on the state-graph correspondence. The state machine detection module is used to input the process operation image acquired at the current moment into a pre-trained target recognition model to obtain the target recognition result, and input the target recognition result into the state machine. The state machine determines the corresponding state to enter based on the target recognition result and each state transition event: when a first state transition event is detected, the corresponding first state is entered and the first state is recorded. Other non-first state transition events directly enter the corresponding state. When a second state transition event is detected based on the first state, the second state is entered and the second state is recorded. Other non-second state transition events enter the corresponding state. The step detection module is used to check whether the process operation steps meet the standards based on the state transition change information of the state machine: when entering the final state or after the time period is reached, it is determined whether the state change occurs in the order of state transition change. If not, the state is determined to be an operation violation. If the state machine state does not change within the specified time window, the operation is determined to be untimely. The state transition change information refers to the changes of each state of the state machine from startup to entering the final state.

7. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the steps of the process operation step inspection method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the process operation step inspection method as described in any one of claims 1 to 5.

Images