SOP compliance detection method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of information intelligence, in particular to an SOP compliance detection method and system, electronic equipment and a storage medium, and the SOP compliance detection method comprises the steps: obtaining a continuous original video stream, carrying out the feature extraction of a sampling frame in the video stream according to a time sequence, and converting the feature into a first feature vector, thereby achieving the lightweight feature extraction, and improving the detection precision. A basis is provided for subsequent action response; in combination with the obtained feature difference between the adjacent sampling frames, an operation node is captured to determine a key frame, the calculation load is greatly reduced, and redundant frames are abandoned; according to the method, key frame information is subjected to structured output, semantic matching is performed on the key frame information and an SOP knowledge base of structured storage, the whole process from video input to matching result output can be completed in an extremely short time, and the real-time response requirements of scenes such as an industrial production line or safety monitoring and the like are met.

Description

Technical Field

[0001] This invention relates to the field of information intelligence technology, and in particular to a SOP compliance testing method, system, electronic device, and storage medium. Background Technology

[0002] In manufacturing, healthcare, education, and workplace safety, Standard Operating Procedures (SOPs) are crucial for ensuring operational consistency, safety, and quality control. Traditional SOP monitoring relies primarily on manual observation, sensor signals, or offline video analysis, which has significant limitations in terms of real-time performance, semantic understanding, and automation.

[0003] Current mainstream methods mainly include: Sensor signal detection: Detecting whether an operation has occurred by installing infrared, pressure, or torque sensors; Video feature change detection: using optical flow, color histograms, or feature differences to determine the point of action; Action classification based on deep learning: identifying predefined action categories using CNN or 3D-CNN; Template-match-based process detection: matching the operation sequence according to manually defined rules or action sequence templates.

[0004] While these methods may be effective in specific scenarios, they are difficult to meet the requirements of industrial-grade intelligent monitoring in open task environments. Summary of the Invention

[0005] The purpose of this invention is to provide a SOP compliance detection method, system, electronic device, and storage medium to solve the problem that existing industrial-grade intelligent monitoring cannot adapt to the need for rapid response, and further solve the problem that it is difficult to meet the requirements of industrial-grade intelligent monitoring when facing unknown actions, complex scenarios, and real-time feedback needs.

[0006] The technical solution of this invention is: a SOP compliance detection method, comprising: Acquire a continuous raw video stream, extract features from the sampled frames in the video stream in time sequence, and convert them into a first feature vector; The feature differences between adjacent sampling frames are compared to capture operation nodes and determine key frames; wherein, the feature differences are obtained by comparing the first feature vectors between adjacent sampling frames. The keyframe information is output in a structured manner and mapped to action events arranged in chronological order; the action events arranged in chronological order are semantically matched with a pre-stored SOP knowledge base; wherein the pre-stored SOP knowledge base is expressed in a structured manner.

[0007] Preferably, the step of extracting features from the sampled frames in the video stream in time sequence and converting them into a first feature vector includes: The CLIP model is used to map the sampled frames in the acquired original video stream to a high-dimensional semantic vector space; wherein, the sampled frame is each frame in the video stream, or a frame sampled at a preset frequency; Each of the sampled frames is converted into a first feature vector, wherein the first feature vector includes a semantic feature vector.

[0008] Preferably, the comparison to obtain feature differences between adjacent sampled frames includes: Obtain the first feature vector corresponding to adjacent sampling frames, wherein the adjacent sampling frames are collected in time sequence, including the sampling frame at the previous time moment and the sampling frame at the next time moment; Based on the first feature vectors of the previous sampling frame and the next sampling frame, the distance between the first feature vectors of adjacent sampling frames is obtained; Define a feature difference threshold, and define the sampling frame at the next time step when the distance between the first feature vectors of adjacent sampling frames is greater than the feature difference threshold as a key frame.

[0009] Preferably, the keyframe information structured output includes using a zero-sample action recognition model to map the keyframe to the visual language embedding space established by the CLIP model, in order to identify the action semantics corresponding to the keyframe, output the action semantics with the highest confidence and its corresponding first confidence, and form the first structured data.

[0010] Preferably, the step of using a zero-shot action recognition model to map keyframes to the visual language embedding space established by the CLIP model to identify the action semantics corresponding to the keyframes, outputting the action semantics with the highest confidence and its corresponding first confidence, forming the first structured data, includes: Construct the complete query text and convert it into a text feature vector; Obtain the second feature vector of the keyframe, the second feature vector including the image feature vector; The image feature vectors of the keyframes are semantically similar to all the text feature vectors to obtain a first semantic matching score; The text feature vector with the highest similarity to the second feature vector of the keyframe is determined, and the corresponding query text is used as the action semantic output. At the same time, the first semantic matching score between the second feature vector of the keyframe and the text feature vector with the highest similarity is defined as the first confidence level.

[0011] Preferably, the structured output of the keyframe information further includes detecting key objects using an open-vocabulary visual localization model, outputting the object label with the highest confidence and its corresponding second confidence, forming second structured data.

[0012] Preferably, the first structured data and the second structured data are fused to obtain structured keyframe information; the structured keyframe information includes action semantics, object labels, and a comprehensive confidence score of the interaction behavior obtained based on the first confidence score of the action semantics and the second confidence score of the object labels.

[0013] Preferably, the keyframe information is output in a structured manner and mapped to action events arranged in a time sequence, including: The action semantics, object labels, and overall confidence scores of keyframes arranged in chronological order are sequentially mapped online to execution steps; wherein, the execution steps are the corresponding action events.

[0014] Preferably, the sequentially arranged action events are semantically matched and compared with a pre-stored SOP knowledge base, including: Each standard action step in the SOP knowledge base is transformed into a semantic feature vector to obtain a standard action vector. The structured keyframe information corresponding to the action events arranged in chronological order is transformed into semantic feature vectors to obtain the recognition action vectors. For each arriving action event, its corresponding identified action vector is semantically matched with all standard action vectors in the SOP knowledge base to determine the execution steps at the current moment.

[0015] Preferably, each of the time-ordered action events is semantically similar to the pre-stored SOP knowledge base to obtain a corresponding second semantic matching score. When all the second semantic matching scores are lower than the preset action recognition threshold, the corresponding action event is judged as non-compliant behavior and recorded as an alarm event.

[0016] According to another aspect of this application, this application also discloses an SOP compliance detection system for performing the above-described SOP compliance detection method.

[0017] According to another aspect of this application, this application also discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the SOP compliance detection method.

[0018] According to another aspect of this application, a storage medium is also disclosed for storing a computer program, which, when executed, implements the SOP compliance detection method.

[0019] Compared with the prior art, the advantages of the present invention are: It acquires continuous raw video streams, extracts features from sampled frames in the video stream in time sequence, and converts them into first feature vectors, enabling lightweight processing and providing a foundation for subsequent action responses. By combining the feature differences between adjacent sampled frames, it captures operation nodes to determine key frames, significantly reducing computational load and discarding redundant frames. The structured output of key frame information is semantically matched with a structured SOP knowledge base, enabling the entire process from video input to matching result output to be completed in a very short time, meeting the real-time response requirements of scenarios such as industrial production lines or safety monitoring. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of one embodiment of the SOP compliance testing method described in this invention; Figure 2 This is a flowchart of another embodiment of the SOP compliance testing method described in this invention; Figure 3 This is a block diagram of the SOP compliance testing system described in this invention. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to specific embodiments: Standard Operating Procedures (SOPs) are standardized operating procedures and requirements for a specific event, presented in a uniform format to guide and regulate daily work. Their key objectives are to ensure consistency and standardization, guarantee quality and compliance, and clearly define responsibilities for traceability. They can be applied to, but are not limited to, the following fields: manufacturing (equipment operation, product assembly, quality monitoring on production lines); healthcare (surgical preparation, drug management, medical device management); laboratories (experimental operations, equipment calibration, sample handling); aviation (takeoff checks, air traffic control procedures, ground maintenance); and food service (food preparation, hygiene and cleaning, customer service procedures).

[0022] This application provides a computer system comprising a terminal device and a server. The terminal device has a client installed and running, and the server provides backend services to the client. The SOP compliance detection method of this application can be executed by the terminal device, by the server, or by both the terminal device and the server.

[0023] In some embodiments, the terminal device can be any electronic device product that can interact with the user; for example, the terminal device can be a mobile phone, a computer (e.g., a personal computer, a desktop computer, a handheld computer, a tablet computer), a wearable device, a smart TV, a smart speaker, etc.

[0024] In some embodiments, the terminal device includes a data acquisition terminal and a display terminal. The core function of the data acquisition terminal is to capture and sense physical information (e.g., video, images, data), convert it into digital signals, and send it to the server. It can be a network camera, analog camera, USB camera, industrial camera, action camera, drone, smartphone, tablet computer, industrial control computer, etc. The core function of the display terminal is to receive information from the server (e.g., analysis results, alarms, data) and present it to the user, typically serving as the output interface. It can be a large monitoring screen, video wall, computer, workstation, smartphone, tablet, network control console, alarm indicator light, audible and visual alarm, AR / VR glasses, etc.

[0025] Terminal device can refer to one of multiple terminal devices. In actual application, there can be only one or multiple devices, without limitation.

[0026] A server can be a single server or a server cluster consisting of multiple servers; the server and terminal devices communicate directly or indirectly through wired or wireless networks. A server has data receiving, data processing, and data sending capabilities.

[0027] In some embodiments, the raw video stream can be acquired through a capture terminal, and the server performs comparative analysis on the raw video stream. Finally, the comparative analysis results generated by the server are sent to the display terminal in real time. The capture terminal and the display terminal can be the same terminal device or different terminal devices.

[0028] The following describes the SOP compliance detection method provided in this application embodiment, with the server as the execution subject, using exemplary applications and implementations of the server provided in the embodiments of this application.

[0029] In one implementation, such as Figure 1 As shown, a method for SOP compliance testing is disclosed. This method relies on computer program implementation and includes the following steps: S101. Obtain a continuous raw video stream, extract features from the sampled frames in the video stream in time sequence, and convert them into a first feature vector; In some embodiments, the original video stream is a sequence of image frames arranged sequentially in time, each frame being a complete still image. It can be a pixel array that has been compressed, encoded, and decoded for the server to read, typically in RGB or YUV format.

[0030] Timing refers to the sequential relationship between frames. The essence of the original video stream not only contains visual information, but also the dynamic information of how this visual information changes over time.

[0031] A sampled frame is a single static image frame extracted from the original video stream according to certain rules for analysis; the frame rate of the video is usually 25 frames per second or 30 frames per second, and the changes between consecutive frames are very small.

[0032] Feature extraction refers to the process of extracting representative and key information from sampled frames; these features can be visual features, including color distribution, texture, shape contour, etc.

[0033] The first feature vector refers to the result of representing a "feature" using a vector composed of a series of numerical values. This is the output of the feature extraction process, which digitizes the image content of the sampled frame into a mathematically expressed vector. Each dimension of the vector represents a certain latent feature of the sampled frame. Sampled frames with similar content will have their first feature vectors close in distance in the vector space, for example, using Euclidean distance or cosine distance.

[0034] Understandably, acquiring a continuous raw video stream, extracting features from sampled frames in the video stream in time sequence, and converting them into a first feature vector; transforming the massive, unstructured, and difficult-to-process raw video stream into a compact and computable mathematical representation vector, makes it possible to achieve lightweighting, while also providing a foundation for subsequent action response.

[0035] S102. Compare and obtain the feature differences between adjacent sampling frames, capture operation nodes and determine key frames; wherein, the feature differences are obtained by comparing the first feature vectors between adjacent sampling frames; In some embodiments, adjacent sampling frames are two frames that are extracted consecutively in chronological order in a video stream. They can also be understood as two frames that are immediately adjacent in a time-sequential sampling sequence. For example, when the sampling frequency is 1 fps, the frame at second T and the frame at second T+1 are "adjacent sampling frames".

[0036] Feature difference is the distance or difference between the first feature vectors of adjacent sampled frames. Since the first feature vector is a numerical array representing the image content of the corresponding sampled frame, feature difference uses mathematical methods to quantify the degree of dissimilarity between two numerical arrays. By converting the degree of change in visual content into a quantifiable numerical value, the larger this value, the smaller the similarity between adjacent sampled frames.

[0037] An operation node can be defined as a time marker, determined by the feature differences between adjacent sampled frames, representing the starting point of an action or a significant change in state within a video stream. It is not a frame, but a moment in time. When an operator performs a specific, discrete action, it causes a drastic change in the scene's visual information within the video stream over a short period. This change results in extremely low similarity between the first feature vectors of two adjacent sampled frames. For example, in an industrial setting, an operator "reaching for a screwdriver" creates a distinct peak in the sequence of feature differences, and the operation node is the moment corresponding to this peak.

[0038] A keyframe is a sampled frame acquired at an operation node. It captures a stable state of an action or the most representative moment of the action. Based on the acquisition of keyframes, a very small number of keyframes can represent the entire lengthy video stream, while carrying the visual information that needs to be analyzed.

[0039] Understandably, by comparing the feature differences between adjacent sampled frames, capturing operation nodes, and determining keyframes, the computational load can be significantly reduced. The server only performs subsequent complex action recognition and localization analysis on the keyframes, discarding redundant frames, which is crucial for achieving real-time processing. The final output keyframes are a sparse, time-ordered sequence.

[0040] S103. Output the keyframe information in a structured manner and map it into action events arranged in chronological order; perform semantic matching between the action events arranged in chronological order and the pre-stored SOP knowledge base; wherein the pre-stored SOP knowledge base adopts a structured expression.

[0041] In some embodiments, structuring is used to transform raw, unorganized data into a uniform, structured format that is easy for the server to process and understand. That is, it transforms the scattered, disorganized information identified in keyframes into a pre-defined, well-organized data object, such as JSON, which contains structured JSON data.

[0042] Mapping is the process of establishing a correspondence between elements in one set and elements in another set. It is a data processing procedure that aims to convert structured keyframe information into standardized action events.

[0043] A Standard Operating Procedure (SOP) knowledge base is a container for storing knowledge about standard operating procedures. It uses a structured representation, typically described in JSON or a graph structure; for example: {"step": 1, "action": "pick up tool"}, {"step": 2, "action": "align parts"}, {"step": 3, "action": "tighten screw"}, {"step": 4, "action": "inspect result"}.

[0044] The SOP knowledge base is stored in SON or graph structure, supports natural language input for generation and rapid modification, and is suitable for rapid migration and deployment across different production lines or tasks.

[0045] Understandably, the keyframe information is structured and output, and mapped to action events arranged in time sequence; the action events arranged in time sequence are semantically matched with a pre-stored SOP knowledge base; the entire process from video input to matching result output can be completed in a very short time, meeting the real-time response requirements of scenarios such as industrial production lines or safety monitoring.

[0046] In summary, the method provided in this disclosure acquires a continuous raw video stream, extracts features from sampled frames in the video stream in a time sequence, and converts them into a first feature vector, making lightweighting possible and providing a foundation for subsequent action responses. By combining the feature differences between adjacent sampled frames, it captures operation nodes to determine key frames, significantly reducing computational load and discarding redundant frames. The structured output of key frame information is semantically matched with a structured SOP knowledge base, enabling the entire process from video input to matching result output to be completed in a very short time, meeting the real-time response requirements of scenarios such as industrial production lines or safety monitoring.

[0047] In another implementation, such as Figure 2 As shown, a method for SOP compliance testing is disclosed. This method relies on computer program implementation and includes the following steps: S201. Obtain a continuous raw video stream, and use the CLIP model to map the sampled frames in the obtained raw video stream to a high-dimensional semantic vector space. Each sampled frame is converted into a first feature vector; the first feature vector includes a semantic feature vector.

[0048] In some embodiments, the CLIP (Contrastive Language-Image Pre-training) model can simultaneously understand images and text, align them in the same semantic space, and ultimately output a high-dimensional floating-point vector. For example, the CLIP model typically outputs a 512-dimensional or 768-dimensional vector. Semantically similar images will have their first feature vectors very close in space, for example, using cosine distance / cosine similarity.

[0049] S202. Obtain the first feature vector corresponding to the adjacent sampling frames. The adjacent sampling frames are collected in time sequence, including the previous sampling frame and the next sampling frame. Based on the first feature vector of the previous sampling frame and the next sampling frame, obtain the distance between the first feature vectors of the adjacent sampling frames. Define a feature difference threshold. Define the next sampling frame corresponding to the distance between the first feature vectors of the adjacent sampling frames that is greater than the feature difference threshold as a key frame.

[0050] In some embodiments, the first feature vectors of all time-sequentially arranged sampled frames are arranged sequentially, for example: V1, V2, V3, V4, V5, ..., Vn; where V1 is the first feature vector of the first sampled frame, and Vn is the first feature vector corresponding to the last sampled frame. Starting from the second first feature vector in the arrangement, it is compared with the first feature vector of the previous one in turn, thereby obtaining the distance between the first feature vectors of adjacent sampled frames; for example, comparing V2 (the sampled frame at a later time step) with V1 (the sampled frame at a previous time step) can obtain the distance between the first feature vectors of two sampled frames V1 and V2; comparing V3 (the sampled frame at a later time step) with V2 (the sampled frame at a previous time step) can obtain the distance between the first feature vectors of two sampled frames V2 and V3. It should be noted that "the sampled frame at a previous time step" and "the sampled frame at a later time step" are always relative to two adjacent sampled frames. For each sampled frame, its relative to the previous sampled frame is defined as the "sampled frame at a later time step", and its relative to the next sampled frame is defined as the "sampled frame at a previous time step".

[0051] In some embodiments, the distance between the first feature vectors between adjacent sampling frames is defined as the cosine distance. The cosine distance measures the difference in direction between the two first feature vectors, without regard to the absolute magnitude between the two vectors, and is suitable for measuring the semantic changes of the vectors.

[0052] In some embodiments, the distance between the first feature vectors of adjacent sampling frames can also be defined as cosine similarity. The difference between cosine similarity and cosine distance is that the range of cosine similarity is between -1 and 1, where "1" indicates that they are exactly the same and "-1" indicates that they are exactly opposite. Cosine distance = 1 - cosine similarity, with a range of between 0 and 2, where "0" indicates that they are exactly the same and "2" indicates that they are exactly opposite.

[0053] In other words, the semantic similarity between any two first feature vectors can be represented by cosine distance or cosine similarity. In this application, since the distance between the first feature vectors of adjacent sampling frames is calculated, taking cosine distance as an example, n-1 cosine distance differences can be calculated between n first feature vectors.

[0054] In some embodiments, a feature difference threshold is defined, which is any value within the range of cosine distance, for example, a feature difference threshold of 0.8. The subsequent sampling frame corresponding to the point where the distance between the first feature vectors of adjacent sampling frames is greater than the feature difference threshold is defined as a keyframe. For example, the cosine distance between (V1, V2) is 0.1, between (V2, V3) is 0.4, between (V3, V4) is 0.3, and between (V4, V5) is 1.2. Since the cosine distance between (V4, V5) is greater than the feature difference threshold of 0.8, it proves that the subsequent sampling frame (V5) has undergone a significant change compared to the previous sampling frame (V4). Therefore, the sampling frame corresponding to V5 is defined as a keyframe.

[0055] S203. Using a zero-sample action recognition model, keyframes are mapped to the visual language embedding space established by the CLIP model to identify the action semantics corresponding to the keyframes, output the action semantics with the highest confidence and its corresponding first confidence, and form the first structured data.

[0056] In some embodiments, a complete query text is constructed and converted into a text feature vector. Constructing the complete query text involves defining a list of actions and converting it into a natural language description. The action list can be defined based on a pre-stored SOP knowledge base, such as "pick up parts" or "align parts." Since the CLIP model is trained using natural language descriptions, rich context can enhance its understanding; therefore, "pick up parts" can be enriched with a natural language description, such as "a worker picking up a strip part."

[0057] In some embodiments, a complete query text is constructed, which includes fine-grained standard steps in the entire industrial process. A text feature vector, for example, a 512-dimensional vector, is output for each query text using the CLIP model. This results in a set of text feature vectors, T1, T2, T3, T4, ..., Tn, where each text feature vector corresponds to a specific textual description of an action.

[0058] In some embodiments, a second feature vector of the keyframe is obtained, which includes an image feature vector; the keyframe is output as an image feature vector, for example, a 512-dimensional vector, using the CLIP model. Each keyframe also corresponds to a set of image feature vectors, P1, P2, P3, P4, ..., Pn.

[0059] In some embodiments, the image feature vector of each keyframe is semantically similar to all text feature vectors to obtain a first semantic matching score. For example, the similarity between the image feature vector P1 and each text feature vector T1, T2, T3, T4, ..., Tn is calculated. This similarity can be defined as cosine similarity, which measures the degree of proximity of two vectors in a direction. The higher the value, the more similar they are. A score list S1, S2, S3, S4, ..., Sn can then be obtained. These scores are the first semantic matching scores.

[0060] In some embodiments, the text feature vector with the highest similarity to the second feature vector of the keyframe is determined, and the corresponding query text is output as the action semantics; at the same time, the first semantic matching score between the second feature vector of the keyframe and the text feature vector with the highest similarity is defined as the first confidence level.

[0061] For example, the text feature vector with the highest similarity to the second feature vector of the keyframe is T2, and the cosine similarity / first semantic matching score is 0.85. In this case, the query text corresponding to T2 is "a worker picking up a strippart", which is output as the action semantics, and the first confidence score is 0.85.

[0062] S204. Use an open-vocabulary visual localization model to detect key objects, output the object label with the highest confidence and its corresponding second confidence, forming the second structured data.

[0063] In some embodiments, similar to step S203, a query list is constructed using an open-vocabulary visual localization model. The list contains object names, and each object name corresponds to an object label. The appearance features of each region in the keyframe are matched with the object names in the query list based on similarity. Finally, the region with the highest matching degree and its corresponding object label are selected, and their similarity is output as a second confidence score.

[0064] In some embodiments, all candidate regions and all object labels of the keyframe are traversed. A candidate region is a region in the keyframe where an object may exist. The object label with the highest similarity for each candidate region is found. For example, if the object label with the highest similarity for a certain candidate region is "screwdriver" with a similarity of 0.95, then the object label is screwdriver and the second similarity is 0.95.

[0065] In this application, based on steps S203 and S204, an open-vocabulary visual model and a zero-shot action recognition model are introduced. The system can recognize new actions, tools, or objects that have not appeared before without retraining for a specific process, significantly reducing model maintenance costs. Furthermore, based on lightweight feature extraction of keyframes and the aforementioned asynchronous parallel architecture, the entire process from video input to alarm output can be completed within 200 milliseconds, meeting the real-time response requirements of scenarios such as industrial production lines or security monitoring. Moreover, by mapping keyframes to the visual language embedding space established by the CLIP model, the action semantics corresponding to the keyframes can be identified, enabling the recognition of complex semantic anomalies such as sequential errors, missing steps, and erroneous execution.

[0066] S205. The first structured data and the second structured data are fused to obtain structured keyframe information. The structured keyframe information includes action semantics, object labels, and a comprehensive confidence of the interaction behavior obtained based on the first confidence of action semantics and the second confidence of object labels.

[0067] In some embodiments, the first structured data includes: {"frame": 105, "action": "a worker picking up a strip part", "confidence": 0.85}; The second type of structured data includes: {"frame": 105, "Object label": "screwdriver", "confidence": 0.95}; At this point, the overall confidence level can be calculated using the average value algorithm: (0.85 + 0.95) / 2 = 0.9. The final structured keyframe information is as follows: {"frame": 105, "action": "a worker picking up a strip part", "Objectlabel": "screwdriver", "confidence": 0.9}.

[0068] S206. The action semantics, object labels, and overall confidence scores of the keyframes arranged in chronological order are sequentially mapped online to execution steps; wherein, the execution steps are the corresponding action events.

[0069] In some embodiments, the structured data based on keyframes corresponds to a specific execution step. For example, in frame 105, based on the action semantics and object labels in the structured data, the execution step is "a worker picking up a screwdriver," which is a complete and meaningful action event that has been identified.

[0070] S207. Perform semantic feature vector conversion on each standard action step in the SOP knowledge base to obtain a standard action vector; perform semantic feature vector conversion on the structured keyframe information corresponding to the action events arranged in chronological order to obtain an identification action vector; for each arriving action event, perform semantic matching between its corresponding identification action vector and all standard action vectors in the SOP knowledge base to determine the execution step corresponding to the current moment.

[0071] In some embodiments, both the standard action vector and the recognized action vector can be obtained using the CLIP model, thereby vectorizing the standard actions in each SOP knowledge base and the recognized actions in the keyframe information. The cosine similarity between the current recognized action vector and each standard action vector in the standard action vector library is calculated, and the standard action vector with the highest similarity is found. The step in the SOP knowledge base corresponding to this standard action vector is the execution step corresponding to the current moment.

[0072] In some implementations, it is necessary not only to find the standard action vector that is closest to each identified action vector, but also to determine whether the corresponding identified action is compliant. Therefore, each action event arranged in chronological order is semantically matched with a pre-stored SOP knowledge base to obtain a corresponding second semantic matching score. When all second semantic matching scores are lower than a preset action recognition threshold, the corresponding action event is judged as non-compliant behavior and recorded as an alarm event.

[0073] In some embodiments, when an alarm event occurs, an alarm needs to be issued immediately, which can be done through audible and visual alerts, interface prompts, or network message pushes. Simultaneously, the alarm event, timestamp, keyframe screenshots, and anomaly type should be recorded. Anomaly types generally include: incorrect action sequence, repeated actions, missing necessary objects, presence of incorrect tools / parts, and abnormal location.

[0074] Other embodiments of this application also disclose a detection system for performing SOP compliance detection methods, such as... Figure 3 As shown, it includes a video acquisition module, a keyframe detection module, a structured data output module, and a comparison module.

[0075] The video capture module is used to capture raw video streams, which contain a sequence of continuously arranged image frames, each of which is a complete still image.

[0076] The keyframe detection module extracts sampled frames from the original video stream, performs feature extraction, and captures operation nodes to determine keyframes based on the feature differences between adjacent sampled frames. Since the frame rate of video is typically 25 or 30 frames per second, the changes between consecutive frames are very small. In practical applications, if the sampled frames were every single frame in the video stream, keyframes could easily be lost due to the minute changes between each frame, and the data processing volume would be enormous. Therefore, the sampled frames are defined as frames sampled from the original video stream at a preset frequency, making it easier to capture keyframes based on the changes between adjacent sampled frames. The feature difference is the distance between the feature vectors of adjacent sampled frames. By digitizing the image content of the sampled frames into mathematically expressed vectors, the differences between all adjacent sampled frames are quantified, and keyframes can be determined by comparison based on a preset feature difference threshold.

[0077] The structured data output module includes a zero-shot action recognition model and an open-vocabulary visual localization model. The zero-shot action recognition model maps keyframes to the visual language embedding space established by the CLIP model to identify the action semantics corresponding to the keyframes, outputting the action semantic with the highest confidence and its corresponding first confidence, forming the first structured data. The open-vocabulary visual localization model detects key objects, outputting the object label with the highest confidence and its corresponding second confidence, forming the second structured data. The first and second structured data can be fused to obtain structured keyframe information; the structured keyframe information includes action semantics, object labels, and a comprehensive confidence of the interaction behavior obtained based on the first confidence of the action semantics and the second confidence of the object labels.

[0078] The comparison module maps structured keyframe information arranged chronologically into action events online and performs semantic feature vector conversion to obtain recognized action vectors. It also performs semantic feature vector conversion on each standard action step in the SOP knowledge base to obtain a standard action vector. The comparison module compares each recognized action vector with the standard action vector to determine the execution step corresponding to the current moment. Simultaneously, based on a preset action recognition threshold, it determines whether the similarity between the current recognized action vector and the standard action vector is lower than the threshold, thereby judging whether the currently executed action event is compliant.

[0079] In some embodiments, the SOP compliance detection system also includes an alarm push module, which is used to issue an alarm when a non-compliant action event occurs. The alarm push module can issue an alarm through audible and visual warnings, interface prompts, network message pushes, and simultaneously record the alarm event, timestamp, keyframe screenshots, and anomaly type.

[0080] This application provides an electronic device in which the processor reads computer-executable instructions from a computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the SOP compliance detection method described above in this application.

[0081] This application also provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the training method or text processing method of the text vectorization model provided in this application.

[0082] In the above description, the term "some embodiments" refers to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0083] In the above description, the terms "first" and "second" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application can be implemented in an order other than that shown in the figures or descriptions.

[0084] The above embodiments are merely illustrative of the technical concept and features of the present invention, intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and should not be construed as limiting the scope of protection of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all changes falling within the meaning and scope of the equivalents of the claims are intended to be included within the present invention.

Claims

1. A method for detecting SOP compliance, characterized in that, include: Acquire a continuous raw video stream, extract features from the sampled frames in the video stream in time sequence, and convert them into a first feature vector; The feature differences between adjacent sampling frames are compared to capture operation nodes and determine key frames; wherein, the feature differences are obtained by comparing the first feature vectors between adjacent sampling frames. The keyframe information is output in a structured manner and mapped to action events arranged in chronological order; the action events arranged in chronological order are semantically matched with a pre-stored SOP knowledge base; wherein the pre-stored SOP knowledge base is expressed in a structured manner.

2. The SOP compliance testing method according to claim 1, characterized in that, The step of extracting features from sampled frames in the video stream in time sequence and converting them into a first feature vector includes: The CLIP model is used to map the sampled frames in the acquired original video stream to a high-dimensional semantic vector space; wherein, the sampled frame is each frame in the original video stream, or a frame sampled at a preset frequency; Each of the sampled frames is converted into a first feature vector, wherein the first feature vector includes a semantic feature vector.

3. The SOP compliance testing method according to claim 1, characterized in that, The comparison to obtain feature differences between adjacent sampled frames includes: Obtain the first feature vector corresponding to adjacent sampling frames, wherein the adjacent sampling frames are collected in time sequence, including the sampling frame at the previous time moment and the sampling frame at the next time moment; Based on the first feature vectors of the previous sampling frame and the next sampling frame, the distance between the first feature vectors of adjacent sampling frames is obtained; Define a feature difference threshold, and define the sampling frame at the next time step when the distance between the first feature vectors of adjacent sampling frames is greater than the feature difference threshold as a key frame.

4. The SOP compliance testing method according to claim 3, characterized in that, The keyframe information structured output includes using a zero-sample action recognition model to map keyframes to a visual language embedding space established by the CLIP model in order to identify the action semantics corresponding to the keyframes, output the action semantics with the highest confidence and its corresponding first confidence, and form the first structured data.

5. The SOP compliance testing method according to claim 4, characterized in that, The zero-shot action recognition model maps keyframes to the visual language embedding space established by the CLIP model to identify the action semantics corresponding to the keyframes, outputs the action semantics with the highest confidence and its corresponding first confidence, forming the first structured data, including: Construct the complete query text and convert it into a text feature vector; Obtain the second feature vector of the keyframe, the second feature vector including the image feature vector; The image feature vectors of the keyframes are semantically similar to all the text feature vectors to obtain a first semantic matching score; The text feature vector with the highest similarity to the second feature vector of the keyframe is determined, and the corresponding query text is used as the action semantic output. At the same time, the first semantic matching score between the second feature vector of the keyframe and the text feature vector with the highest similarity is defined as the first confidence level.

6. The SOP compliance testing method according to claim 5, characterized in that, The keyframe information structured output also includes using an open vocabulary visual localization model to detect key objects, outputting the object label with the highest confidence and its corresponding second confidence, forming the second structured data.

7. The SOP compliance testing method according to claim 6, characterized in that, The first structured data and the second structured data are fused to obtain structured keyframe information; the structured keyframe information includes action semantics, object labels, and a comprehensive confidence score of the interaction behavior obtained based on the first confidence score of the action semantics and the second confidence score of the object labels. This involves outputting keyframe information in a structured manner and mapping it to action events arranged in a time sequence, including: The action semantics, object labels, and overall confidence scores of keyframes arranged in chronological order are sequentially mapped online to execution steps; wherein, the execution steps are the corresponding action events.

8. The SOP compliance testing method according to claim 7, characterized in that, The sequentially arranged action events are semantically matched and compared with a pre-stored SOP knowledge base, including: Each standard action step in the SOP knowledge base is transformed into a semantic feature vector to obtain a standard action vector. The structured keyframe information corresponding to the action events arranged in chronological order is transformed into semantic feature vectors to obtain the recognition action vectors. For each arriving action event, its corresponding identified action vector is semantically matched with all standard action vectors in the SOP knowledge base to determine the execution steps at the current moment.

9. A SOP compliance detection system, characterized in that, Used to perform the SOP compliance testing method according to any one of claims 1-8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the SOP compliance detection method according to any one of claims 1-8.

Images