Intelligent operation coordination control method and system

Abstract

This invention relates to the field of intelligent manufacturing and production management technology, specifically to an intelligent operation collaborative control method and system. The method includes: acquiring real-time video streams of the work area; calling a pre-trained neural network model to parse the video streams to identify fine-grained behavioral sequences of operators; comparing the identified behavioral sequences with pre-stored standard operating procedures to generate compliance judgment results; and generating and outputting corresponding guidance information to the work area when the judgment result indicates abnormal behavior. Furthermore, the method also includes analyzing behavioral data to generate on-site management optimization suggestions and quantitatively evaluating the effectiveness of the optimization measures. This invention achieves precise real-time monitoring and guidance of the operation process and enables on-site optimization and effect evaluation based on objective data, effectively improving the level of operation standardization, product quality consistency, and overall production efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing and production management technology, specifically to an intelligent operation collaborative control method and system. Background Technology

[0002] In industrial production, assembly, and testing scenarios, ensuring the standardization and compliance of operating procedures is crucial for guaranteeing product quality, improving production efficiency, and reducing safety risks. Traditional operation management and collaborative control primarily rely on paper-based work instructions, regular on-site inspections, and workers' personal experience and self-discipline. With the development of industrial automation and informatization, simple on-duty / off-duty detection systems based on video surveillance and production management systems that confirm the completion of steps by scanning barcodes or RFID have emerged. However, these existing technological solutions have significant limitations. First, they mostly remain at the level of coarse-grained monitoring of "whether a person is on duty" or "whether a step has been triggered," unable to identify and judge the operator's specific "action posture," "operation sequence," and "process details" with fine-grained precision. For example, they cannot detect whether screws are tightened at the correct angle and in the correct sequence, which is often a key factor affecting product reliability and consistency. Second, existing guidance methods are mostly pre-training or post-correction, lacking the ability to provide real-time, precise, and contextualized guidance during operation. They cannot achieve "learning by doing," resulting in long training cycles for new employees and an inability to intervene immediately in the errors of skilled workers. Furthermore, although concepts such as lean manufacturing and 6S management emphasize improving efficiency through optimizing workplace layout and tool placement, traditional optimization methods heavily rely on managers' subjective experience and lack objective, quantifiable data support. This makes it difficult to accurately assess the true impact of a specific improvement measure (such as adjusting tool positions) on actual operational efficiency and production cycle time. Therefore, there is an urgent need in this field for an intelligent collaborative control solution that can deeply integrate behavior recognition, real-time guidance, and data-driven optimization analysis to solve the entire chain of problems from accurate perception and intelligent decision-making to continuous optimization.

[0003] Therefore, the existing technology still needs further development. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide an intelligent operation collaborative control method and system to solve the problems existing in the prior art.

[0005] To achieve the above-mentioned technical objectives, according to a first aspect of the present invention, the present invention provides an intelligent operation collaborative control method, comprising: S1. Obtain the real-time video stream of the target work area; S2. Call the pre-trained neural network model to parse the real-time video stream in order to identify the behavior sequence of the operators; S3. Compare the behavior sequence with the pre-stored standard operating procedures to generate a compliance judgment result; S4. When the compliance judgment result indicates abnormal behavior, generate and output corresponding guidance information to the target work area.

[0006] Specifically, the step of calling a pre-trained neural network model to parse the real-time video stream to identify the behavioral sequences of the workers includes: The real-time video stream is segmented frame by frame or segment by segment to extract image data containing the operator's actions; the image data is input into the neural network model, and the neural network model outputs a classification of at least one action and the spatial location information of the key points of the action; the classification and the spatial location information of the key points are associated with time sequence to form a behavioral sequence representing the operation process.

[0007] Specifically, comparing the behavioral sequence with pre-stored standard operating procedures to generate a compliance judgment result includes: The standard operating procedure defines a standard behavior sequence, which includes the type of standard operation action, the sequential relationship between standard operation actions, and the spatial location range of standard key points corresponding to each standard operation action. The operation action type and sequence in the identified behavior sequence are matched with the standard behavior sequence, and it is determined whether the spatial location information of the key points is within the spatial location range of the standard key points of the corresponding standard operation action. Based on the matching and judgment results, the compliance of the current operation in at least one dimension of action type, action sequence, and action posture is determined.

[0008] Specifically, generating and outputting corresponding guidance information to the target work area includes: Based on the type of anomaly indicated by the compliance judgment result, match the corresponding corrective action instructions or standard operating procedure demonstration materials from the guidance information database; convert the corrective action instructions into an audio signal and play it through audio equipment deployed in the target work area; and / or push the standard operating procedure demonstration materials in the form of graphics or animation to the display terminal visible to the workers in the target work area.

[0009] Specifically, the method further includes: While outputting the guidance information, the system continuously performs behavior recognition and compliance judgment based on the latest real-time video stream. If the behavior sequence of the operator is identified as corrected to conform to the standard operating procedure within a preset time, the current guidance information is stopped being output. If the behavior sequence is not corrected after a preset time, the guidance information level is upgraded or an early warning is triggered.

[0010] Specifically, the method also includes on-site management analysis based on behavior recognition data, including: Record and statistically analyze behavioral data and timestamps related to tool picking and placing, material transfer, and personnel movement in the behavioral sequence; based on the behavioral data and timestamps, analyze at least one of the following results: standard tool seeking time, invalid movement paths between workstations, and the occupation of operating space by material stacking; based on the analysis results, generate management optimization suggestions for work site layout, tool positioning, or cleaning and tidying.

[0011] Specifically, the management optimization suggestions generated based on the analysis results for work site layout, tool placement, or cleaning and tidying include: When the average time for tool positioning exceeds a threshold, a suggestion is generated to adjust the tool's storage location to a more easily accessible area; when frequent invalid cross-workstation movements are identified, a suggestion is generated to re-plan the location of adjacent workstation processes or material storage areas; when video stream analysis detects debris unrelated to the current operation on the workbench, a real-time voice or light prompt is generated to clean it up.

[0012] Specifically, the method also includes efficiency analysis, including: Based on behavior recognition data within historical periods, the total duration of effective operational behaviors is calculated to obtain the workstation operation efficiency benchmark value. The changes in the workstation operation efficiency benchmark value before and after the implementation of on-site management optimization suggestions in different periods are correlated to assess the specific impact of various on-site management optimization measures on operation efficiency, so as to determine the priority of optimization measures.

[0013] Specifically, the changes in workstation operational efficiency baselines before and after the implementation of the on-site management optimization suggestions at different times are used to assess the specific impact of each on-site management optimization measure on operational efficiency, including: Establish a management measure library, with each management measure associated with the category of on-site elements it affects; track the implementation of measures, monitor changes in behavioral data corresponding to relevant on-site elements through continuous behavior identification, and calculate the improvement in the benchmark value of work efficiency caused by the implementation of the measure; rank multiple optimization suggestions in the same period or region according to the improvement, and feed the ranking results back to the management system.

[0014] According to a second aspect of the present invention, an intelligent operation collaborative control system is provided, comprising: The visual acquisition module is configured in the target work area to acquire real-time video streams; The behavior recognition module, connected to the visual acquisition module, is used to call a pre-trained neural network model to parse the real-time video stream and identify the behavior sequence of the operator; The compliance judgment module, connected to the behavior recognition module, is used to compare the behavior sequence with the pre-stored standard operating procedures and generate a compliance judgment result. The intelligent guidance module, connected to the compliance judgment module, is used to generate and output guidance information to the target work area when the judgment result indicates abnormal behavior; The on-site management analysis module, connected to the behavior recognition module, is used to analyze behavioral data related to on-site organization in the behavior sequence and generate management optimization suggestions.

[0015] Beneficial effects: The intelligent operation collaborative control method and system provided by this invention brings multi-dimensional and multi-level beneficial effects by constructing a complete closed loop of perception, analysis, decision-making and intervention.

[0016] At the quality control and standardized operation level, this invention, through the deployment of a deep neural network model, achieves fine-grained recognition of worker operation actions. It can accurately analyze micro-behaviors such as tool-holding posture and component assembly angles, and compare them in real time with digitized standard operating procedures. When the system detects deviations in operation type, sequence, or posture, it can provide immediate and precise guidance and correction through multimodal methods (such as voice and screen graphics). This "immediate error correction" capability significantly shifts the quality control point from final inspection to each operational step, effectively eliminating quality risks caused by non-standard operations, significantly improving first-pass yield and product consistency, while reducing reliance on workers' long-term experience and focus, and shortening the training cycle for new employees.

[0017] In terms of production efficiency and process optimization, this invention transcends the scope of traditional monitoring, transforming behavioral recognition data into valuable on-site analysis resources. The system automatically records and analyzes the time and paths consumed by actions such as tool retrieval, material handling, and personnel movement. Through data mining techniques, it objectively reveals irrationalities and waste in on-site layout and tool placement, and automatically generates data-supported management optimization suggestions. Furthermore, by defining and calculating key efficiency indicators such as the "percentage of effective operation time," the system can quantitatively evaluate the specific effects of each optimization measure (such as adjusting tool rack positions) before and after implementation. This allows for the scientific prioritization of improvement measures, guiding managers to invest resources in the most efficient areas and driving data-driven continuous lean improvement on the production floor.

[0018] At the level of human-machine collaboration and intelligent management, this invention achieves flexible and adaptable collaborative control. The system's guidance mechanism is not a mechanical alarm, but an intelligent interaction with state perception and feedback capabilities. It can automatically stop prompting after the operator corrects the error in time, and escalate the warning level if the problem persists. This anthropomorphic interaction mode enhances the friendliness and acceptability of human-machine collaboration. Ultimately, this invention integrates discrete perception, control, and management functions into an organic whole, not only controlling the operational behavior of "people" but also optimizing the entire production system of "people, machines, materials, methods, and environment," forming an intelligent collaborative control closed loop from real-time operation guidance to macro-process optimization, providing solid technical support for enterprises to upgrade to intelligent manufacturing. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the intelligent operation collaborative control method provided in a specific embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments in this application, other similar embodiments obtained by those skilled in the art without creative effort should all fall within the scope of protection of this application. Furthermore, directional terms mentioned in the following embodiments, such as "up," "down," "left," and "right," are only for reference to the directions in the accompanying drawings; therefore, the directional terms used are for illustrative purposes and not for limiting the invention.

[0021] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments.

[0022] Please see Figure 1 This invention provides an intelligent operation collaborative control method, comprising: S1. Obtain the real-time video stream of the target work area.

[0023] It should be further explained that this method first uses multiple high-definition network cameras (1080p resolution, 25fps frame rate) deployed above the target work area (such as assembly station, welding station) to simultaneously acquire real-time video streams covering the entire work space without blind spots. The video streams are then transmitted to the edge computing server via a network switch.

[0024] S2. Call the pre-trained neural network model to parse the real-time video stream in order to identify the behavior sequence of the operators.

[0025] It should be further noted that the pre-trained neural network model is a deep convolutional neural network that incorporates temporal modeling, and its training process specifically includes: (1) Data collection: Skilled workers operate according to standard operating procedures, record videos through multi-angle cameras, and simultaneously record the action category, key point coordinates (obtained through motion capture system or manual annotation) and corresponding timestamps to form a training dataset; (2) Model construction: I3D (Dilated 3D Convolutional Network) is used as the backbone network. The input is a continuous 16-frame image segment extracted from the video stream (the image size is adjusted to 224×224 pixels). The network output consists of two branches. The first branch is the action classification score vector. ,in Branch 2 is the prediction of the two-dimensional coordinates of 17 joints of the human body, which is the total number of defined standard operation action categories (such as "pick up the screw", "align", "tighten", "put down the screwdriver" etc.). ; (3) Model training: using the cross-entropy loss function Supervision action classification branches, among which It's a true one-hot tag for the action. It is the first of the predicted score vectors Each component; using the smoothed L1 loss function. Supervision of key point detection branch, among which It is the predicted number Coordinates of each joint point These are the actual coordinates; the total loss is Among them, the balance factor The value was set to 0.5, and through extensive experiments, it was found that this value could balance the learning of the two tasks. The Adam optimizer was used, with an initial learning rate of 0.001 and a batch size of 8. The training was conducted for 200 epochs on 4 NVIDIA Tesla V100 GPUs.

[0026] S3. Compare the behavior sequence with the pre-stored standard operating procedures to generate a compliance judgment result.

[0027] It should be further explained that the behavioral sequence is compared with a pre-stored standard operating procedure (SOP). The SOP is digitized into a structured state-action diagram, where each state node is associated with a permitted standard action and its parameter range. For example, the standard parameters for the "screw in" action include: the angle θ between the screwdriver axis and the normal to the workpiece surface is in the range of [-3°, 3°], and the tightening duration is 2 ± 0.5 seconds. The compliance judgment result is based on the Dynamic Time Warping (DTW) algorithm to calculate the similarity between the real-time behavioral sequence and the standard sequence, and checks whether the key parameters are generated within the specified range. Specifically, if the DTW distance... If all parameters are within the specified range, the system is considered compliant; otherwise, it is considered abnormal. The Dynamic Time Warping (DTW) distance threshold is also considered. The value is set based on the average distance of historical compliant operation sequences plus twice the standard deviation, with a typical value of 1.5.

[0028] S4. When the compliance judgment result indicates abnormal behavior, generate and output corresponding guidance information to the target work area.

[0029] It should be further explained that the guidance information is generated based on a predefined "abnormality type - guidance content" mapping library. For example, if the abnormality type is "tool angle exceeds limit", the mapped guidance content is an animation highlighting the correct angle and a voice text saying "Please adjust the tool angle to keep it vertical". This voice text is converted into an audio stream by a TTS engine (such as iFlytek engine) on the edge server and played through a directional speaker next to the workstation. At the same time, the guidance animation is pushed to the 21.5-inch industrial touch screen in front of the workstation through an industrial switch and displayed as an overlay on the electronic SOP interface.

[0030] Understandably, this method constructs a complete intelligent control closed loop from perception, understanding, decision-making to intervention. It utilizes deep neural networks to achieve precise analysis of complex, fine-grained operational actions, surpassing traditional "on-the-job inspection" and capable of recognizing microscopic movements such as screwdriver grip and tightening angle. By transforming unstructured video streams into structured behavioral sequence data that can be precisely compared with digital standard operating procedures (SOPs), the computer can judge the compliance of each operational step like an experienced technician. Once a deviation is detected, the system can immediately provide precise guidance through multimodal methods (visual and auditory), achieving "immediate error correction." This shifts quality control from final product inspection to every production step, significantly reducing quality defects and rework costs caused by human error, improving first-pass yield and production consistency, and reducing over-reliance on individual worker experience and attention.

[0031] Specifically, the step of calling a pre-trained neural network model to parse the real-time video stream to identify the behavior sequence of the operator includes: segmenting the real-time video stream frame by frame or segment by segment to extract image data containing the operator's operation actions; inputting the image data into the neural network model, and having the neural network model output a classification of at least one operation action and key point spatial location information of the operation action; and forming a behavior sequence representing the operation process based on the temporal sequence association of the classification and the key point spatial location information.

[0032] It should be further explained that the method specifically includes: (1) Video preprocessing and segmentation: After receiving the video stream, the edge computing server first uses the YOLOv5s-based object detection model (which has been pre-trained with industrial scene images containing various tooling and workstations) to perform human detection and locate bounding boxes for each frame of the image. ,in The coordinates of the bounding box center are The width and height were then set, and the bounding box was subsequently enlarged by 1.3 times (to include the full range of motion of the limbs) before cropping out the image of the person's region. Next, a sliding window approach is used, dividing the frame into segments of 16 frames (stride=8), starting from consecutive frames... Constructing video clips , which serves as the basic input unit for behavior recognition models.

[0033] (2) Action classification and key point detection: video clips The input is fed into the I3D dual-branch network. The action classification branch outputs one... A 3D vector, after passing through a Softmax layer, gives the segment belonging to... Probability distribution of each action category Choose the category with the highest probability. This serves as the predicted action for the segment. The keypoint detection branch outputs heatmaps of 17 keypoints, and by finding the maximum value of each heatmap, a set of two-dimensional coordinates for the 17 keypoints is obtained. These key points include the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Specifically, to identify tools, we attached a small tool key point detection head near the wrist key point to predict the coordinates of the tool (such as the tip of a screwdriver). .

[0034] (3) Temporal correlation and sequence generation: For a continuous operation process (such as tightening a screw), a series of ordered video clips will be generated. and their corresponding prediction results To smooth out noise, we use a sliding window of length 5 to perform voting filtering on the action category sequence and a Kalman filter to smooth the keypoint coordinate sequence. Finally, the action sequence is formally represented as a list of tuples sorted by timestamps: ,in It is the first The center timestamp of each segment.

[0035] Understandably, this step provides a complete and feasible technical path from raw video to structured behavior sequences. Object detection and segmentation effectively focus on the workers, eliminating interference from complex backgrounds. A dual-branch network is used for simultaneous action classification and keypoint detection, not only identifying "what action is being performed" but also accurately obtaining "body posture and tool position," providing crucial data for subsequent posture compliance judgment. Temporal modeling techniques (sliding window voting, Kalman filtering) are used for post-processing of the original recognition results, effectively overcoming the jitter and misjudgments that may exist in single-frame recognition, generating stable, smooth behavior sequences with temporal context. This sequence serves as a bridge connecting raw perception with high-level semantic understanding (compliance judgment), and its accuracy and robustness are the cornerstone of the entire system's reliable operation.

[0036] Specifically, the step of comparing the behavior sequence with a pre-stored standard operating procedure (SOP) to generate a compliance judgment result includes: the SOP defines a standard behavior sequence, which includes the type of standard operation action, the sequential relationship between standard operation actions, and the spatial location range of standard key points corresponding to each standard operation action; matching the operation action type and sequence in the identified behavior sequence with the standard behavior sequence, and determining whether the key point spatial location information is within the spatial location range of the standard key points of the corresponding standard operation action; and determining the compliance of the current operation in at least one dimension of action type, action sequence, and action posture based on the matching and judgment results.

[0037] It should be further explained that the digital definition of the Standard Operating Procedure (SOP) adopts a directed graph-based structure. For example, a "screw installation" procedure can be defined as: State S0 (Ready) — [Action A1: Take screw], State S1 — [Action A2: Take screwdriver], State S2 — [Action A3: Align screw hole], State S3 — [Action A4: Screw in], State S4 (Completed). Each standard action node contains the following attributes: (a) Expected action type, such as "screw in" for A4; (b) Precursor and successor actions define sequence constraints, such as A4 must follow A3; (c) Spatial range of standard key points, for example, for the action of "screwing in a screw", the standard posture is defined as: the height difference between the left and right wrists. Pixels (ensuring two-handed coordination), the deviation distance between the screwdriver key point and the estimated screw hole position (obtainable through prior calibration). Pixels, and screwdriver axis vector with the workpiece surface normal vector The included angle .

[0038] Furthermore, the comparison process is performed in real time: when the behavior recognition module outputs the current action classification... and key point set At that time, the compliance assessment module first checks Does it match the expected action type desired by the current SOP state? If it matches, proceed to attitude determination and calculate the above. and The value is used to determine whether the action is within the standard range. If the action type matches but the sequence is incorrect (e.g., "screw in" is detected but the current state expects "align with screw hole"), a compliance judgment result of "incorrect sequence" is generated. If the action type does not match, an "action error" result is generated. The compliance judgment result is a structure, for example: {isCompliant:false,errorType:"posture_error",errorDetail:{parameter:"angle",actualValue:8.5,standardRange:[0,5]},timestamp:t_n}.

[0039] Understandably, this step represents a leap from behavioral "recognition" to "evaluation," directly reflecting quality control. By transforming standard operating procedures (SOPs) into a computable graphical model containing multi-dimensional constraints (type, sequence, spatial orientation), computers can perform precise and objective automated inspections. In particular, the quantitative judgment of spatial orientation (the range of key point locations) can capture potential problems where the action type is correct but the operation method is not standardized (e.g., tilting a screw may lead to stripped threads or insufficient torque), something traditional visual inspection or barcode scanning cannot achieve. This refined compliance judgment ensures that product quality depends not only on "taking the correct steps," but also on "completing each step in the correct way," fundamentally improving product consistency and reliability, and demonstrating a high degree of professionalism and creativity.

[0040] Specifically, generating and outputting corresponding guidance information to the target work area includes: matching the corresponding corrective action instructions or standard operation demonstration materials from the guidance information database according to the anomaly type indicated by the compliance judgment result; converting the corrective action instructions into a voice signal and playing it through audio equipment deployed in the target work area; and / or pushing the standard operation demonstration materials in the form of graphics or animation to the display terminal visible to the workers in the target work area.

[0041] It should be further explained that the guidance information database is a relational database table, whose core fields include: exception type code, guidance level, text prompt, animation / image resource path, and voice text. For example, for a record with exception type code "POSTURE_ANGLE_DEVIATION", its guidance level is "Warning", the text prompt is "Tool angle exceeds standard range", the animation resource path points to a 3D animation file (.mp4 format) demonstrating the correct vertical downward pressing operation, and the voice text is "Please note, please keep the tool perpendicular to the workpiece surface". The matching process is as follows: when the compliance judgment module generates a result containing errorType: "posture_error" and errorDetail:{parameter: "angle"...}, the system will query the guidance information database using errorType and parameter as the joint key to obtain the corresponding record.

[0042] Furthermore, the conversion and output process is as follows: The system's main control program (running on an edge server) calls a text-to-speech (TTS) service to synthesize the spoken text content, presented in a Chinese female voice, at a medium speed (default parameters) into a WAV format audio data stream. This stream is then sent to the network audio player (such as a speaker column with an amplifier) ​​corresponding to the workstation via a real-time audio streaming protocol (such as RTSP) or directly via AOIP (AudiooverIP), driving the speaker to play the audio in real time. Simultaneously, the main control program loads the corresponding animation / image resources from a file server and overlays them onto the current interface of the electronic SOP Web application running on the workstation's touchscreen via a WebSocket protocol or a video streaming protocol (such as RTMP). The system can configure the output mode according to the guidance level; for example, a "warning" level triggers both voice and animation, while a "hint" level may only trigger screen highlighting and flashing.

[0043] Understandably, this step is the core interactive link for achieving "real-time intervention" and "learning by doing." It's not simply an error alarm, but a precise, context-sensitive auxiliary system. By matching guidance content from a structured information database, the accuracy and consistency of guidance information are ensured. The use of TTS technology allows for dynamic generation of voice prompts, offering high flexibility. Combining visual guidance (animation / text and images) with auditory guidance (voice) conforms to best practices in multimodal human-computer interaction, adapts to noisy industrial environments (workers may wear earplugs), and provides redundant information channels to ensure effective reception of guidance information. This immediate and precise guidance effectively interrupts the continuation of erroneous operations, guiding workers to correct them immediately, nipping potential quality defects in the bud, and significantly reducing training and error correction costs.

[0044] Specifically, the method further includes: while outputting the guidance information, continuously performing behavior recognition and compliance judgment based on the latest real-time video stream; if the behavior sequence of the operator is identified as having been corrected to conform to the standard operating procedure within a preset time, then the current guidance information is stopped being output; if the behavior sequence is still not corrected after the preset time, then the guidance information level is upgraded or an early warning is triggered.

[0045] It should be further explained that this step implements a closed-loop control logic with feedback. After the system is triggered, an internal timer is started, with a timer duration of [duration missing]. This is a configurable parameter, with a preferred value of 8 seconds. The choice of 8 seconds is based on ergonomic considerations: from receiving guidance information to understanding, reacting, and executing corrective actions, the average worker needs 3-5 seconds. Allowing an 8-second margin is sufficient to cover most situations while avoiding disruption to the work cycle due to excessive waiting time. During the time period, the system continuously performs video analysis. The logic for determining "corrected" is: at any point in time after the guidance is triggered, the latest action category identified by the system is consistent with the action category expected in the current SOP, and the latest calculated key point parameters (such as angle) are consistent. ,deviation Once the correction is detected, the system immediately sends a "Stop Booting" control command to the audio player and display screen, terminating the current playback / display of voice and animation, and the process automatically returns to the normal SOP prompt state. If the timer expires (i.e., more than 8 seconds) and no correction is detected, the upgrade logic is triggered.

[0046] Further upgrade strategies include: (1) Upgrade guidance: Upgrade from "Warning" level to "Severe Warning" level, for example, increase the voice volume by 20%, slow down the speech rate, and make the tone more serious, while the animation on the screen changes to a bright red warning border flashing; (2) Warning reporting: The system sends a warning message to the Production Execution System (MES) or the team leader's mobile terminal (such as PAD) via the MQTT protocol. The message format is JSON, containing {"station_id":"A01","worker_id":"W123","error":"angle_deviation","duration":8.5","status":"not_fixed"}, requesting manual intervention.

[0047] Understandably, this step endows the system with "intelligence" and "flexibility," making its behavior more akin to that of an experienced on-site supervisor. It doesn't mechanically and statically issue alarms, but dynamically adjusts its behavior based on real-time feedback from operators (manifested as changes in their movements). This state-based feedback mechanism avoids unnecessary interference with corrected errors, ensuring smooth operation, while also allowing for decisive escalation when problems persist, preventing issues at a single workstation from impacting the efficiency of the entire production line or causing batch quality incidents. This tiered, adaptive intervention strategy significantly improves the efficiency and user-friendliness of human-machine collaboration, representing a key aspect of the advanced nature of intelligent control systems.

[0048] Specifically, the method also includes analyzing the work site based on behavioral recognition data, including: recording and statistically analyzing behavioral data and timestamps related to tool picking and placing, material transfer, and personnel movement in the behavioral sequence; analyzing at least one of the following based on the behavioral data and timestamps: standard tool seeking time, invalid movement paths between workstations, and the occupancy of operating space by material stacking; and generating management optimization suggestions for work site layout, tool positioning, or cleaning and tidying based on the analysis results.

[0049] It should be further explained that the method specifically includes: (1) Data Recording: The system establishes a behavior event log table in the background. Whenever a specific behavior is identified, a record is generated. For example, when the action is identified as "picking up tools" and the key point of the tool moves from the tool rack area to the operation area, the event Event_tool_pick is recorded: {type: "tool_pick", tool_id: "screwdriver_ph2", timestamp: t1, from_location: "rack_A1"}. When the displacement of the "body torso center point" exceeds the threshold (e.g., 20 pixels) for multiple consecutive frames, it is determined as "personnel movement", and the starting point, ending point coordinates, and duration are recorded. The workbench area is continuously monitored through background subtraction or semantic segmentation models. When a connected area of ​​non-tools and non-current process materials is detected that is greater than the threshold, the system will detect the movement of the tool. (This threshold corresponds to the projected area of ​​a small part) and the static time exceeds If the time is less than 1 second, it is recorded as a "clutter accumulation" event.

[0050] (2) Data analysis: Tool positioning time The calculation formula is: ,in It is the time when the tool is picked up. This refers to the start time of the tool's intended action, which can be approximated by the end time of the previous action or the time it takes for the operator's head to turn towards the tool rack. The system tracks all actions taken by each tool within the past shift (e.g., 8 hours). average and standard deviation Invalid movement paths are identified by analyzing personnel movement trajectories. These trajectories are then rasterized, and the frequency of access to raster cells that do not belong to standard process paths (predefined) is counted. Frequently accessed non-standard paths are marked as "invalid paths." Material stacking is quantified by calculating the frequency and average duration of "debris accumulation" events per unit time.

[0051] (3) Suggestion Generation: Based on the analysis results, the system automatically generates natural language descriptions of suggestions in the daily report. For example, if a certain tool... If the time is less than 1 second (empirical threshold), the following message will be generated: "The average seek time of the tool [electric screwdriver] is too long ( It is currently stored on the upper shelf of the tool cabinet behind you. It is recommended to move it to the first shelf of the tool rack on the right front. "If a high-frequency invalid path is detected from workstation A to material area B, then the following message will be generated: 'Frequent back-and-forth travel has been detected from the assembly station to the material storage area, with a daily cumulative distance of approximately 150 meters. It is recommended to add a small turnover box for this material on the right side of the assembly station.'" Understandably, this step extends the system's value from "real-time quality control" to "production process optimization" and "lean management." Utilizing massive amounts of time-series data generated by behavioral recognition, and through data mining techniques, it objectively and quantitatively reveals hidden wastes (Muda) in the production floor, such as "searching," "unnecessary movement," and "poor organization." These analytical results and recommendations provide data-driven decision-making support for on-site managers in 6S improvement, layout optimization, and tool placement management. This gives continuous improvement (Kaizen) activities clear, quantifiable goals and directions, propelling production management from empiricism to scientific and refined approaches, and significantly enhancing the system's overall value and creative potential.

[0052] Specifically, the management optimization suggestions generated based on the analysis results for work site layout, tool placement, or cleaning and tidying include: when the average time for tool location exceeds a threshold, generating a suggestion to adjust the tool's storage location to a more easily accessible area; when frequent invalid cross-workstation movements are identified, generating a suggestion to re-plan the locations of adjacent workstation processes or material storage areas; and when video stream analysis detects debris unrelated to the current operation on the workbench, generating real-time voice or light prompts for cleaning.

[0053] It should be further explained that this step is the specific execution and real-time application of the previous data analysis step, and its specific design includes: (1) Logic for generating tool placement optimization suggestions: The system maintains a spatial model of a "golden operating area" for each workstation. This area is usually centered on the center of the operator's torso, with a radius of... cm, height is A curved space within a cm radius represents the most comfortable ergonomic operating range. The system calculates the three-dimensional Euclidean distance from the location of each tool (obtained through tool identification and scene map mapping) to the center of this "golden area." Threshold for average tool location time Set to either of two conditions: a) a fixed empirical threshold of 3.5 seconds; b) a dynamic threshold, which is 1.5 times the average seek time of all tools at this station. If a certain tool's... The system explicitly states in its optimization suggestions: "It is recommended to move tool [T] from its current position [P_curr] (distance from the golden area)." Move [P_alt] (cm) to the alternative location [P_alt] (distance) (cm). The alternative location is selected from the currently unoccupied standard tool position that is closer.

[0054] (2) Logic for generating layout and logistics optimization suggestions: Define a judgment rule for "frequent invalid moves": in the statistical period The number of invalid paths (derived from trajectory analysis) with the same starting point S and ending point E occurring within one hour. And the path length Meters. When the conditions are met, the system analyzes the functions of points S and E (e.g., S is the operation point of workstation A, and E is the material rack B), and then queries the process flow chart to determine whether this movement is outside the standard process. If so, it generates a suggestion: "Frequent unnecessary material handling has been detected between [S] and [E]. It is recommended to adjust the position of [material rack B] closer to [workstation A], or evaluate setting up a line-side warehouse for this material next to [workstation A]." (3) Real-time 6S supervision function implementation: The system runs a lightweight semantic segmentation model (such as DeepLabV3+MobileNet) to segment the workbench area in the video frame in real time, distinguishing between "current process materials", "tools", and "miscellaneous items / background". When the area of ​​the identified "miscellaneous items" exceeds the threshold, the system will implement a real-time 6S supervision function. And it continues to exist for more than If the timer reaches a certain threshold, real-time supervision is triggered. The system illuminates a flashing yellow light on the programmable indicator light (tri-color light) at the workstation and plays a pre-recorded prompt through the workstation's voice unit: "Please tidy up your work surface promptly." This supervision continues until video analysis shows that the "cluttered" area has disappeared or its area has decreased below a certain threshold.

[0055] Understandably, this step transforms the results of data analysis into direct, actionable instructions, even achieving automated real-time supervision. Tool optimization suggestions combine time and spatial data, making them more persuasive. Layout optimization suggestions analyze high-frequency movement paths in conjunction with process logic, tracing the "phenomenon" to the "possible cause," thus improving the accuracy of the suggestions. The real-time 6S supervision function is particularly important; it changes the traditional periodic inspection model, enabling routine and immediate management of "sorting, setting in order, cleaning, and sweeping," helping to cultivate good habits (discipline) among employees. These functions work together to make the system not only a monitoring and guidance tool, but also a proactive, data-driven on-site management optimization engine, achieving "collaborative control" of the work environment itself.

[0056] Specifically, the method also includes efficiency analysis, including: calculating the total time ratio of effective operation behaviors based on behavior recognition data within historical periods to obtain a workstation operation efficiency benchmark value; associating the changes in workstation operation efficiency benchmark values ​​before and after the implementation of on-site management optimization suggestions in different periods to assess the specific impact of various on-site management optimization measures on operation efficiency, so as to determine the priority of optimization measures.

[0057] It should be further explained that the benchmark value for workstation operation efficiency The calculation formula is: in, It is the total duration of the selected historical period, such as an 8-hour work shift. It is the number of all standard operating actions (or "value-added actions") identified within the period. It is the first The "effective operation time" for each standard operating action is defined as the time interval from when the action is identified as starting (e.g., "picking up a screwdriver") to when the action is identified as ending and deemed "compliant". Non-value-added time, such as time spent searching for tools, waiting, moving, or rework, is not included in the numerator. The system automatically calculates the effective operation time for each workstation daily. The system records the values ​​and stores them in the database. Once a management optimization suggestion (such as "location of mobile tool A") is marked as "implemented," the system tracks its implementation for a period of time (e.g., one week). Values, calculate their average. And the average value of the same period before implementation (e.g., the previous week). Compare the results. Determine the improvement in computational efficiency. : In addition, to assess significance, the system can calculate its relative percentage increase. The system establishes a "measure-effect" relationship table to record each measure and its corresponding effect. and When resources are limited and it is necessary to prioritize optimization measures, managers can follow the order of priority. or The measures are ranked, and those with the greatest improvement (i.e., the highest return on investment) are promoted to other similar work positions first.

[0058] Understandably, this step embodies the core principles of lean management—measuring improvement effectiveness with data. By defining a key performance indicator (KPI) of "percentage of effective operation time," the abstract concept of "efficiency" is quantified, making the efficiency of different workstations and different times comparable. Changes in efficiency indicators ( By establishing causal relationships between improvements and specific management measures (such as layout adjustments and tool positioning), objective and quantitative evaluation of the effectiveness of improvement activities is achieved. This breaks through the traditional management dilemma of relying on intuition and difficulty in evaluating improvement effects. Prioritizing optimization measures based on quantitative evaluation results ensures that limited improvement resources (such as engineering time and modification costs) are invested in areas that best enhance overall efficiency, driving the scientific and efficient operation of the Continuous Improvement Cycle (PDCA) and ultimately achieving continuous improvement in the overall efficiency of the production system.

[0059] Specifically, the changes in workstation operational efficiency benchmarks before and after the implementation of on-site management optimization suggestions at different times are associated with the assessment of the specific impact of each on-site management optimization measure on operational efficiency. This includes: establishing a management measure library, with each management measure associated with the category of on-site elements it affects; tracking the implementation of measures, monitoring changes in behavioral data corresponding to relevant on-site elements through continuous behavior identification, and calculating the improvement in operational efficiency benchmarks caused by the implementation of the measures; ranking multiple optimization suggestions from the same period or region based on the improvement, and feeding the ranking results back to the management system.

[0060] It should be further explained that the management action library is a relational database table. Each record contains the following fields: Action ID, Action Description, Implementation Workstation, Implementation Time, Affected Element Category, and Related Key Behavioral Indicators. The Affected Element Category is an enumerated field with possible values ​​such as: tool accessibility, material handling distance, operating space, and workstation layout. The related key behavioral indicators are used for more granular attribution; for example, for tool accessibility actions, the related indicator is the average tool search time. And the angle of torso twist when picking up tools.

[0061] Furthermore, the evaluation process is as follows: During the pre-set observation period (e.g., 5 working days) after the implementation of the measures, the system not only calculates the overall efficiency baseline value of the workstations. It also simultaneously calculates the values ​​of associated Key Behavioral Indicators (KBIs). For example, assuming that after implementing a measure to "adjust tool rack height," the associated average tool seek time increases from... Decrease to Meanwhile, the efficiency benchmark value from Upgraded to The system will calculate the improvement rate of KBI. and efficiency improvement value To facilitate cross-sectional comparisons of multiple measures, the system employs a comprehensive scoring model, such as calculating the overall benefit score for each measure. : in, and These are the maximum values ​​of efficiency improvement and KBI improvement rate among all evaluated measures during the same period, respectively, used for normalization. and These are weighting coefficients, usually This indicates a greater emphasis on overall efficiency improvement. The system calculates... The values ​​are sorted in descending order to generate a "Top N Most Effective Improvement Measures" list. This list is automatically pushed to production managers, industrial engineers, and others via WeChat bots, emails, or directly into the MES system dashboard, guiding them to prioritize the promotion of high-ranking measures.

[0062] Understandably, this step represents a deepening and intelligentization of efficiency analysis, addressing the question of "how to identify the most effective improvement from multiple options." By establishing a measure library and linking it to key behavioral indicators, the attribution analysis of efficiency improvements becomes more precise, distinguishing between the effectiveness of a specific measure and the impact of other factors (such as improved worker proficiency). The comprehensive scoring model integrates the final result (efficiency improvement) and process indicators (KBI improvements), making the evaluation more comprehensive and robust. Automated sorting and feedback mechanisms transform data insights into management actions, forming a complete data-driven optimization loop from "waste discovery" to "improvement implementation" and then to "evaluating effectiveness and prioritizing promotion." This significantly enhances the intelligence level and decision-making efficiency of continuous improvement in the production system, representing a high-level manifestation of the system's creativity.

[0063] This invention provides another embodiment, which provides an intelligent operation collaborative control system, the intelligent operation collaborative control system comprising: (1) Visual acquisition module, configured in the target work area, used to acquire real-time video stream.

[0064] It should be further explained that the vision acquisition module consists of several industrial cameras (such as Hikvision MV-CH250-10GM), which are deployed at different angles around the workstation and connected to the local area network through a gigabit Ethernet switch. The resolution is set to 1920x1080, the frame rate is set to 25fps, and appropriate lighting sources are provided to ensure image quality.

[0065] (2) Behavior recognition module, connected to the visual acquisition module, used to call the pre-trained neural network model to parse the real-time video stream and identify the behavior sequence of the operator.

[0066] It should be further explained that the behavior recognition module is deployed on an industrial edge computing server, which runs the Ubuntu system and the deep learning inference framework TensorRT. The pre-trained neural network model is converted into TensorRT engine files to accelerate inference. The module pulls video streams from the vision acquisition module via the RTSP protocol, and after parsing, publishes structured behavior sequence data through an internal message queue (such as ZeroMQ).

[0067] (3) Compliance judgment module, connected to the behavior recognition module, used to compare the behavior sequence with the pre-stored standard operating procedure and generate a compliance judgment result.

[0068] It should be further explained that the compliance judgment module runs on the same server or another server. It contains a digital SOP database (such as MySQL) that stores the standard operating procedure diagrams for all workstations. This module subscribes to behavior sequence messages, performs real-time comparisons, and publishes the results.

[0069] (4) Intelligent guidance module, connected to the compliance judgment module, used to generate and output guidance information to the target work area when the judgment result indicates abnormal behavior.

[0070] It should be further explained that the intelligent guidance module includes a guidance content management server and a workstation controller. The guidance content server stores audio and animation materials, and the workstation controller is connected to the speaker and display screen of each workstation. When an abnormal message is received from the compliance judgment module, the workstation controller retrieves the corresponding material from the content server and controls the output device to play it.

[0071] (5) On-site management analysis module, connected to the behavior recognition module, is used to analyze the behavior data related to on-site organization in the behavior sequence and generate management optimization suggestions.

[0072] It should be further explained that the on-site management and analysis module runs on the workshop's data center server. It reads historical behavior data from the logs of the behavior recognition module, runs data analysis algorithms (such as Spark stream processing or periodic batch processing jobs), generates analysis reports and recommendations, and provides them to the MES or management dashboards via a web service interface. All these modules are loosely coupled and connected via a network (industrial Ethernet or 5G private network), exchanging data through well-defined APIs (such as RESTful API or gRPC), collectively forming a distributed, scalable intelligent collaborative control system.

[0073] Understandably, this system, through a modular hardware and software architecture, materializes all the functions of the aforementioned methods, forming a complete technical solution. The visual acquisition module is responsible for high-precision perception; the behavior recognition and compliance judgment module acts as the "intelligent brain" to realize cognition and decision-making; the intelligent guidance module acts as the "executor" to realize precise intervention; and the on-site management and analysis module acts as the "optimization engine" to realize continuous improvement. Each module has a clear division of labor, and through standardized interface communication, it ensures both the system's real-time performance (guidance response in milliseconds) and the depth and breadth of analysis and decision-making. This architecture makes the system easy to deploy, maintain, and expand, adaptable to different application scales from a single workstation to an entire workshop, providing a powerful and implementable collaborative control infrastructure for intelligent manufacturing.

[0074] In a preferred embodiment, this application also provides an electronic device, the electronic device comprising: The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the intelligent job collaborative control method described herein. The computer device can be broadly categorized as a server, terminal, or any other electronic device with the necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, memory, network interface, communication interface, etc., connected via a system bus. The processor of the computer device can be used to provide the necessary computing, processing, and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and internal memory. The non-volatile storage medium may store an operating system, computer programs, etc. The internal memory can provide an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface and communication interface of the computer device can be used to connect and communicate with external devices via a network. When the computer program is executed by the processor, it performs the steps of the method of the present invention.

[0075] This invention can be implemented as a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the steps of the methods of embodiments of the invention to be performed. In one embodiment, the computer program is distributed across multiple network-coupled computer devices or processors, such that the computer program is stored, accessed, and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be executed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be executed by one or more computer devices or processors, and one or more other method steps / operations may be executed by one or more other computer devices or processors. One or more computer devices or processors may execute a single method step / operation, or execute two or more method steps / operations.

[0076] Those skilled in the art will understand that the method steps of this invention can be performed by a computer program instructing related hardware, such as a computer device or processor, to perform the steps of this invention when executed. Depending on the context, any references herein to memory, storage, databases, or other media may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0077] The technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification, provided that such combination does not contain contradictions.

[0078] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An intelligent job coordination control method, characterized by, include: S1. Obtain the real-time video stream of the target work area; S2. Call the pre-trained neural network model to parse the real-time video stream in order to identify the behavior sequence of the operators; S3. Compare the behavior sequence with the pre-stored standard operating procedures to generate a compliance judgment result; S4. When the compliance judgment result indicates abnormal behavior, generate and output corresponding guidance information to the target work area.

2. The intelligent operation collaborative control method according to claim 1, characterized in that, The step of calling a pre-trained neural network model to parse the real-time video stream to identify the behavior sequence of the workers includes: The real-time video stream is segmented frame by frame or segment by segment to extract image data containing the operator's actions; the image data is input into the neural network model, and the neural network model outputs a classification of at least one action and the spatial location information of the key points of the action; the classification and the spatial location information of the key points are associated with time sequence to form a behavioral sequence representing the operation process.

3. The intelligent operation collaborative control method according to claim 2, characterized in that, The step of comparing the behavior sequence with pre-stored standard operating procedures to generate a compliance judgment result includes: The standard operating procedure defines a standard behavior sequence, which includes the type of standard operation action, the sequential relationship between standard operation actions, and the spatial location range of standard key points corresponding to each standard operation action. The operation action type and sequence in the identified behavior sequence are matched with the standard behavior sequence, and it is determined whether the spatial location information of the key points is within the spatial location range of the standard key points of the corresponding standard operation action. Based on the matching and judgment results, the compliance of the current operation in at least one dimension of action type, action sequence, and action posture is determined.

4. The intelligent operation collaborative control method according to claim 1, characterized in that, The process of generating and outputting corresponding guidance information to the target work area includes: Based on the type of anomaly indicated by the compliance judgment result, match the corresponding corrective action instructions or standard operating procedure demonstration materials from the guidance information database; convert the corrective action instructions into an audio signal and play it through audio equipment deployed in the target work area; and / or push the standard operating procedure demonstration materials in the form of graphics or animation to the display terminal visible to the workers in the target work area.

5. The intelligent operation collaborative control method according to claim 4, characterized in that, The method further includes: While outputting the guidance information, the system continuously performs behavior recognition and compliance judgment based on the latest real-time video stream. If the behavior sequence of the operator is identified as corrected to conform to the standard operating procedure within a preset time, the current guidance information is stopped being output. If the behavior sequence is not corrected after a preset time, the guidance information level is upgraded or an early warning is triggered.

6. The intelligent operation collaborative control method according to claim 1, characterized in that, The method also includes on-site management analysis based on behavior recognition data, including: Record and statistically analyze behavioral data and timestamps related to tool picking and placing, material transfer, and personnel movement in the behavioral sequence; based on the behavioral data and timestamps, analyze at least one of the following results: standard tool seeking time, invalid movement paths between workstations, and the occupation of operating space by material stacking; based on the analysis results, generate management optimization suggestions for work site layout, tool positioning, or cleaning and tidying.

7. The intelligent operation collaborative control method according to claim 6, characterized in that, The management optimization suggestions generated based on the analysis results, targeting the layout of the work site, tool placement, or cleaning and tidying, include: When the average time for tool positioning exceeds a threshold, a suggestion is generated to adjust the tool's storage location to a more easily accessible area; when frequent invalid cross-workstation movements are identified, a suggestion is generated to re-plan the location of adjacent workstation processes or material storage areas; when video stream analysis detects debris unrelated to the current operation on the workbench, a real-time voice or light prompt is generated to clean it up.

8. The intelligent operation collaborative control method according to claim 6, characterized in that, The method also includes efficiency analysis, including: Based on behavior recognition data within historical periods, the total duration of effective operational behaviors is calculated to obtain the workstation operation efficiency benchmark value. The changes in the workstation operation efficiency benchmark value before and after the implementation of on-site management optimization suggestions in different periods are correlated to assess the specific impact of various on-site management optimization measures on operation efficiency, so as to determine the priority of optimization measures.

9. The intelligent operation collaborative control method according to claim 8, characterized in that, The changes in baseline workstation operational efficiency before and after the implementation of on-site management optimization suggestions at different times are correlated to assess the specific impact of each on-site management optimization measure on operational efficiency, including: Establish a management measure library, with each management measure associated with the category of on-site elements it affects; track the implementation of measures, monitor changes in behavioral data corresponding to relevant on-site elements through continuous behavior identification, and calculate the improvement in the benchmark value of work efficiency caused by the implementation of the measure; rank multiple optimization suggestions in the same period or region according to the improvement, and feed the ranking results back to the management system.

10. An intelligent operation collaborative control system, characterized in that, The intelligent operation collaborative control method according to any one of claims 1-9 includes: The visual acquisition module is configured in the target work area to acquire real-time video streams; The behavior recognition module, connected to the visual acquisition module, is used to call a pre-trained neural network model to parse the real-time video stream and identify the behavior sequence of the operator; The compliance judgment module, connected to the behavior recognition module, is used to compare the behavior sequence with the pre-stored standard operating procedures and generate a compliance judgment result. The intelligent guidance module, connected to the compliance judgment module, is used to generate and output guidance information to the target work area when the judgment result indicates abnormal behavior; The on-site management analysis module, connected to the behavior recognition module, is used to analyze behavioral data related to on-site organization in the behavior sequence and generate management optimization suggestions.

Images