An assembly operation detection method, device, equipment and medium

Abstract

This invention relates to a method, apparatus, equipment, and medium for detecting assembly operations. It pertains to the field of assembly operation detection technology. The method includes: acquiring a video stream of an assembly work area and determining an image sequence based on the video stream; identifying the assembly operations corresponding to the image sequence to obtain an action sequence; determining the process information and process status corresponding to the action sequence based on the action sequence; and performing a detection operation based on the process information and process status to obtain a target detection result. This invention can automatically monitor and judge the entire assembly process to identify assembly operations, replacing manual assembly control methods and reducing problems such as missed assembly, incorrect assembly, and incorrect process sequence caused by the uncertainty of manual operation.

Description

Technical Field

[0001] This invention relates to the field of assembly operation detection technology, and specifically to a method, apparatus, equipment and medium for detecting assembly operations. Background Technology

[0002] In modern industrial manufacturing, assembly is a key link in product quality control.

[0003] In related technologies, product assembly is often carried out manually. However, due to the uncertainty of manual operation, problems such as omissions, incorrect assembly, and errors in the sequence of processes can easily occur during production, resulting in potential quality risks and rework losses.

[0004] Therefore, there is an urgent need for a method to improve the accuracy of assembly operations. Summary of the Invention

[0005] The technical problem to be solved by this invention is that due to the uncertainty of manual operation, problems such as missing parts, incorrect parts, and incorrect process sequence are prone to occur in production, resulting in quality risks and rework losses.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: a method for detecting assembly operations, the method comprising: acquiring a video stream of an assembly operation area, and determining an image sequence based on the video stream; identifying the assembly operation corresponding to the image sequence to obtain an action sequence; determining the process information and process status corresponding to the action sequence based on the action sequence; and performing a detection operation based on the process information and process status to obtain a target detection result.

[0007] The beneficial effects of this invention are: by performing video acquisition and image analysis on the assembly work area, the assembly operation is automatically identified and an action sequence is formed, thereby determining the corresponding process information and process status. Finally, based on the process information and process status, error prevention detection is performed, which can accurately monitor and judge the entire assembly process in an automated manner, replacing the assembly control method that relies on manual labor, and reducing the problems of missing parts, incorrect parts, and incorrect process sequence caused by the uncertainty of manual operation.

[0008] Based on the above technical solution, the present invention can be further improved as follows.

[0009] Furthermore, the assembly operation corresponding to the image sequence is identified to obtain the action sequence, including: performing target detection on the image sequence to obtain the association information of the image sequence; wherein, the association information includes at least one of hand information, component information, tool information, and assembly platform information; based on the association information, determining the position and category corresponding to the association information; based on the position and category, dividing the assembly operation into a first stage, a second stage, and a third stage; wherein, the first stage is executed before the second stage, and the second stage is executed before the third stage; and based on the first stage, the second stage, and the third stage, determining the action sequence.

[0010] The beneficial effect of adopting the above-mentioned further solution is that by performing target detection on the image sequence to obtain the location and category information of hand information, component information, tool information and assembly platform information, and based on the hand information, component information, tool information and assembly platform information, the assembly operation is divided into three stages, the complete process of the operation is restored, and the accuracy of action recognition is effectively improved.

[0011] Furthermore, based on the first, second, and third stages, an action sequence is determined, including: determining the spatial relationships corresponding to the associated information based on the first, second, and third stages; wherein, the spatial relationship includes at least one of the following: intersection-union ratio, Euclidean distance, relative position, and motion trajectory; based on the spatial relationship, determining the actions to be executed according to preset rules to obtain the action sequence; wherein, the preset rules include: in the first stage, hand information indicates that the hand approaches the part; in the second stage, the intersection-union ratio of the hand information and the part information is greater than the intersection-union ratio threshold; in the third stage, the part information indicates that the part moves with the hand, which is determined as a picking action; in the first stage, hand information indicates that the hand carries the part to approach the assembly platform; in the second stage, the intersection-union ratio of the part and the platform is greater than the intersection-union ratio threshold; in the third stage, the hand leaves the part, which is determined as a placement action; in the second stage, the tool or hand moves vertically downward above the part, which is determined as a pressing action; in the second stage, the tool moves reciprocating horizontally, which is determined as a wiping action.

[0012] The beneficial effect of adopting the above-mentioned further solution is that by calculating the spatial relationships such as the intersection-union ratio, Euclidean distance, relative position and motion trajectory between the hand, parts, tools and platform in stages, and accurately determining assembly actions such as picking up, placing, pressing and wiping according to preset rules, the standardization of actions can be achieved by using temporal and spatial joint reasoning, thereby effectively improving the accuracy of action recognition.

[0013] Furthermore, based on the action sequence, the process information and process status corresponding to the action sequence are determined, including: determining the process information based on the action sequence and a pre-built assembly process standard model; wherein, the pre-built assembly process standard model is used at least to indicate the process action combination; determining the process matching degree between the action sequence and the process action combination based on the action sequence and the process action combination; and determining the process information and process status corresponding to the action sequence based on the process matching degree.

[0014] The beneficial effect of adopting the above-mentioned further solution is that by matching the identified action sequence with the pre-built assembly process standard model and obtaining the process matching degree, it is possible to uniformly map the scattered low-level actions to the process level, accurately locate the currently executed process content and running status, thereby effectively improving the accuracy of action recognition.

[0015] Furthermore, based on process information and process status, inspection operations are performed to obtain target inspection results, including: based on process information and process status, missing assembly inspection, incorrect assembly inspection, and process sequence inspection are performed to obtain the first inspection result corresponding to missing assembly inspection, the second inspection result corresponding to incorrect assembly inspection, and the third inspection result corresponding to process sequence inspection; based on the first inspection result, the second inspection result, and the third inspection result, the target inspection result is obtained.

[0016] The beneficial effect of adopting the above-mentioned further solution is that, based on the determined process information and process status, three detections of missing parts, incorrect parts, and process sequence can be carried out simultaneously. This enables comprehensive and multi-dimensional error prevention judgment of the assembly process, fully covering the problems of missing parts, incorrect parts, and incorrect process sequence that are most likely to occur in manual operation, thereby effectively improving the accuracy of action recognition.

[0017] Furthermore, based on the first detection result, the second detection result, and the third detection result, the target detection result is obtained, including: obtaining the first priority corresponding to the first detection result, the second priority corresponding to the second detection result, and the third priority corresponding to the third detection result; wherein, the second priority is greater than the third priority, and the third priority is greater than the first priority; based on the first priority, the second priority, and the third priority, the first detection result, the second detection result, and the third detection result are filtered to determine the detection result with the highest priority, and this detection result is used as the target detection result.

[0018] The beneficial effect of adopting the above-mentioned further solution is that by setting priorities for missing assembly detection, incorrect assembly detection, and process sequence detection, and filtering the results according to priority, the assembly error with the highest hazard and greatest impact can be prioritized when multiple anomalies occur, thereby improving the safety of assembly.

[0019] Furthermore, the missing item detection includes: determining whether any necessary steps have not been executed or completed based on the process status and the completion rate of necessary actions; the incorrect item detection includes: determining whether the current action is a prohibited action, whether the object being operated on is a permitted object, and whether the placement position is within the preset area; the process sequence detection is: determining whether all the preceding dependent processes of the current process have been completed.

[0020] The beneficial effect of adopting the above-mentioned further solution is that by making targeted judgments on the completion status of necessary processes, the compliance of prohibited actions and objects, and the execution order of preceding dependent processes, a fault-prevention detection logic covering the entire assembly process is formed, which can accurately identify omissions, misassemblies, and errors in the process sequence during assembly.

[0021] Furthermore, the present invention provides a detection device for assembly operations, the device comprising: an acquisition and determination module for acquiring a video stream of an assembly work area and determining an image sequence based on the video stream; an identification module for identifying the assembly operation corresponding to the image sequence to obtain an action sequence; a process determination module for determining process information and process status corresponding to the action sequence based on the action sequence; and a detection execution module for executing a detection operation based on the process information and process status to obtain a target detection result.

[0022] Furthermore, this application provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the detection method for the assembly operation of the first aspect or any corresponding embodiment described above.

[0023] Furthermore, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the detection method for the assembly operation of the first aspect or any corresponding embodiment described above.

[0024] Furthermore, this application provides a computer program product, including computer instructions, which are used to cause a computer to perform an assembly operation of the first aspect described above or any corresponding embodiment thereof, a detection method. Attached Figure Description

[0025] Figure 1 A schematic diagram of the architecture of the assembly operation detection system provided by the present invention; Figure 2 A schematic diagram of the architecture of a detection system for another assembly operation provided by the present invention; Figure 3 A flowchart illustrating an assembly operation detection method provided by the present invention; Figure 4A flowchart illustrating another assembly operation detection method provided by the present invention; Figure 5 A flowchart illustrating another assembly operation detection method provided by the present invention; Figure 6 This is a schematic diagram of process modeling and mapping provided according to an embodiment of the present invention; Figure 7 A flowchart illustrating the detection method for a reassembly operation provided by the present invention; Figure 8 A schematic diagram of a multi-level alarm mechanism method provided according to an embodiment of the present invention. Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0027] In modern industrial manufacturing, the assembly stage is a critical node in product quality control. The assembly process typically involves dozens or even hundreds of steps, each requiring specific operations to be performed in a strict sequence. However, due to the complexity and uncertainty of manual operation, quality problems such as omissions, incorrect assembly, and errors in the sequence of steps are prone to occur during assembly. These problems not only create potential product quality risks but also increase rework costs and extend production cycles. Statistics show that quality problems in the assembly stage account for over 40% of quality losses in manufacturing, with human error accounting for more than 70% of these losses.

[0028] In related technologies, product assembly is often carried out manually. However, due to the uncertainty of manual operation, problems such as omissions, incorrect assembly, and errors in the sequence of processes can easily occur during production, resulting in potential quality risks and rework losses.

[0029] Based on this, the present invention provides a detection method for assembly operations. By performing video acquisition and image analysis on the assembly work area, the assembly operations are automatically identified and action sequences are formed, thereby determining the corresponding process information and process status. Finally, error prevention detection is performed based on the process information and process status. This method can accurately monitor and judge the entire assembly process in an automated manner, replacing the assembly control method that relies on manual labor, and reducing problems such as missing parts, incorrect parts, and incorrect process sequence caused by the uncertainty of manual operation.

[0030] Please refer to Figure 1 The present invention provides a detection system for assembly operations, the system comprising: The system includes: a video acquisition module, an image processing module, a logic judgment module, a data storage module, a human-computer interaction module, and a communication interface module.

[0031] The video acquisition module is used to acquire video streams from the assembly work area and perform preprocessing such as denoising and normalization on the video streams to obtain image sequences.

[0032] The image processing module is used to perform target detection on image sequences, obtain the position and category information of hands, parts, tools, and assembly platforms, and calculate spatial relationships such as intersection-union ratio, Euclidean distance, relative position and motion trajectory between targets based on the time sequence stage, and complete the basic data processing required for action recognition.

[0033] The logic judgment module is used to obtain the action sequence based on spatial relationships and preset rules, determine the process information and process status based on the action sequence and the standard model of assembly process, and perform missing assembly detection, incorrect assembly detection, and process sequence detection according to the process information and process status, and obtain the target detection results according to priority.

[0034] The data storage module is used to store configuration parameters, action sequences, process information, detection results, video clips, and log data.

[0035] The human-computer interaction module is used to configure parameters, display real-time status, show detection results, and provide alarm prompts.

[0036] The communication interface module is used for data interaction with external production systems.

[0037] Please refer to Figure 2 This invention provides another assembly operation detection system, which is used to achieve traceability of assembly operations. The system includes: a data acquisition layer, a data storage layer, a data analysis layer, and a system integration layer.

[0038] At the data acquisition layer, data from the assembly process is collected, which may include: video streams from the assembly work area, key frame images of each process, and high-definition video clips before and after alarms are triggered; at the same time, action sequences, process status, error logs, alarm information, product unique identifier ID, operator information, and execution timestamps are collected and recorded to form a traceable raw dataset.

[0039] At the data storage layer, a combination of local storage and cloud backup is used to store the collected video data and log data in the file system and relational database, respectively.

[0040] At the data analysis layer, multi-level analysis is achieved based on the stored data: First, single-item traceability, which allows querying the complete assembly process, operators, process execution sequence, action records, alarm information and processing results of a product based on its product ID; Second, batch analysis, which statistically analyzes the pass rate, rework rate and error type of the same batch of products; Third, root cause localization, which uses data mining to locate the source of quality problems as process design, operating procedures, equipment status or environmental factors.

[0041] At the system integration layer, the system interfaces with the MES system, ERP system, and quality management system to report the completion status of processes, test results, and alarm information to the upper levels, and receive work orders, product models, and assembly process standards to the lower levels. Based on the test results, the system can send prompts, warnings, or shutdown commands to the equipment control system, thereby achieving interconnection and closed-loop management between the testing system and the factory production system.

[0042] According to an embodiment of the present invention, an embodiment of a detection method for assembly operations is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] This embodiment provides a method for detecting assembly operations, which can be used in an assembly operation detection system. Figure 3 This is a flowchart of a detection method for assembly operations according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps: Step S301: Acquire video streams from the assembly work area and determine image sequences based on the video streams.

[0044] The assembly work area can be understood as the physical workstation area where assembly actions such as picking up, placing, assembling, and clamping product parts are performed. A video stream can be understood as time-series video data composed of multiple frames continuously captured by an industrial camera. An image sequence can be understood as multiple images extracted from the video stream or processed frame by frame and arranged chronologically.

[0045] Specifically, video streams of the assembly work area can be captured by an industrial camera fixedly installed above the assembly station, and then images can be extracted frame by frame to form an image sequence. The resolution of the video stream is no less than 1920×1080, and the frame rate is no less than 30 FPS.

[0046] As an example, a video stream is processed by frame extraction, selecting keyframes at fixed time intervals to obtain an image sequence.

[0047] As an example, the video stream is preprocessed with noise reduction, contrast enhancement, region of interest cropping, and illumination normalization to obtain a clear and stable image sequence.

[0048] Step S302: Identify the assembly operation corresponding to the image sequence to obtain the action sequence.

[0049] Assembly operations can be understood as standardized actions such as picking up, placing, pressing, and wiping performed during the assembly process.

[0050] Specifically, after determining the image sequence, the assembly operations corresponding to the image sequence can be identified to form an action sequence.

[0051] As an example, target detection is performed on the image sequence to identify the categories and locations of hands, parts, tools, and assembly platforms; then, based on the target's positional changes, intersection-over-union ratio, and motion trajectory over time, the sequence is divided into pre-event, during-event, and post-event stages; finally, the action type is determined according to preset rules to form an action sequence arranged by time.

[0052] Step S303: Based on the action sequence, determine the process information and process status corresponding to the action sequence.

[0053] Process information can be understood as the identification information such as the assembly process number, process name, and process content that is currently being performed or has been completed.

[0054] Process status can be used to characterize the progress of process execution, including non-started, in progress, completed, and abnormal states.

[0055] Specifically, after determining the action sequence, the corresponding process information and process status can be determined based on the actions in the action sequence.

[0056] As an example, a standard assembly process model is pre-built, defining the necessary actions, action combinations, and process dependencies for each process. The identified action sequences are then matched against the standard model to calculate the process matching degree. Based on the matching degree, the current process is determined, and the process status is updated. For example, if the action sequence is to pick up a screw, place it, and tighten it, the matched process is the screw fixing process, and the process status is marked as in progress.

[0057] Step S304: Based on the process information and process status, perform the inspection operation to obtain the target inspection result.

[0058] Inspection operations can be understood as missing parts inspection, incorrect parts inspection, and process sequence inspection based on process information and process status.

[0059] The target detection result can be understood as the determination of whether the assembly is qualified, what type of error exists, and the corresponding confidence level.

[0060] Specifically, the detection operation can be determined by process information and process status, and then the target detection result can be obtained based on the detection operation.

[0061] As an example, perform missing parts detection, incorrect parts detection, and process sequence detection; For missing parts detection, the completion rate of necessary actions is used to determine whether there are incomplete processes or missing parts.

[0062] For misassembly detection, it determines whether the action is in violation of regulations, whether the object is used incorrectly, or whether the placement position exceeds the limit.

[0063] For process sequence detection, it determines whether all the preceding dependent processes of the current process have been completed.

[0064] In one possible implementation scenario, a camera captures assembly video of workers, identifying hands, housings, screws, screwdrivers, and the assembly platform frame by frame. Actions such as picking up screws, placing screws, and tightening screwdrivers are identified sequentially to form an action sequence. This action sequence is matched with a preset process model to determine that the current process is screw tightening and is in progress. Subsequently, it checks whether the required actions for this process are complete, whether the screw type is correct, whether the placement position is compliant, and whether it is executed after the previous positioning process is completed. After priority determination, the target detection result with no abnormalities or corresponding error type is output.

[0065] The assembly operation detection method provided in this invention automatically identifies assembly operations and forms an action sequence by performing video acquisition and image analysis on the assembly work area, thereby determining the corresponding process information and process status, and finally performing error prevention detection based on the process information and process status. It can accurately monitor and judge the entire assembly process in an automated manner, replacing the assembly control method that relies on manual labor, and reducing problems such as missing parts, incorrect parts, and incorrect process sequence caused by the uncertainty of manual operation.

[0066] Based on the assembly operation corresponding to the identified image sequence in step S302 of this embodiment, the action sequence is obtained using the following method: Figure 4 The implementation of steps S4021 to S4024 in the above steps is as follows: Step S4021: Perform target detection on the image sequence to obtain the association information of the image sequence; wherein, the association information includes at least one of hand information, component information, tool information and assembly platform information.

[0067] Hand information can be understood as information related to the operator's hand area, posture, and movement status.

[0068] Component information can be understood as the category, model, size, and location of the parts to be assembled.

[0069] Tool information can be understood as information about the type, location, and usage status of the tooling and equipment used in the assembly process.

[0070] Assembly platform information can be understood as the location, area, and relevant information of the assembly operation reference platform.

[0071] Specifically, after determining the image sequence, target detection can be performed on the image sequence to obtain the association information of the image sequence.

[0072] As an example, a convolutional neural network-based object detection model, such as the YOLO series of deep learning object detection networks, can be used. This model is pre-trained using a sample dataset containing hands, various parts, assembly tools, and assembly platforms, enabling it to identify key targets in the assembly scene. During the inference phase, each frame of the image sequence is input into the trained object detection network. The model automatically predicts and outputs the bounding boxes, class probabilities, and confidence scores for hands, parts, tools, and assembly platforms, thus obtaining the corresponding hand, part, tool, and assembly platform information for the image sequence.

[0073] As an example, preprocessing is performed on single-frame images in the image sequence, such as grayscale conversion, filtering and denoising, and binarization, to suppress environmental noise interference and highlight target edges. Then, the external contours of connected regions in the image are obtained through edge detection algorithms or contour extraction algorithms. Suspected target regions are screened based on the contour features of parts and tools. The extracted contours are then matched with pre-established standard part templates and tool templates. The target category and region are determined based on the matching similarity. Finally, the parts and tool regions in the image sequence are identified, forming the corresponding part information and tool information.

[0074] Step S4022: Based on the associated information, determine the location and category corresponding to the associated information.

[0075] Location can be understood as the coordinates, bounding box, or region of the target in the image coordinate system.

[0076] Categories can be understood as types divided according to their target function and purpose, such as hands, screws, housings, screwdrivers, assembly tables, etc. Specifically, the location and category corresponding to the associated information can be obtained by processing the associated information.

[0077] As an example, the pixel positions and regions of hands, parts, tools, and assembly platforms in the image are determined based on the coordinates of the target detection bounding boxes. Based on preset category labels, the targets are categorized into hands, screws, washers, housings, wrenches, screwdrivers, assembly platforms, etc.

[0078] Step S4023: Based on location and category, the assembly operation is divided into a first stage, a second stage, and a third stage; wherein the first stage is executed before the second stage, and the second stage is executed before the third stage.

[0079] The first, second, and third stages can be understood as the pre-assembly preparation stage, the in-process execution stage, and the post-assembly completion stage, respectively, according to the sequence of assembly operations.

[0080] Specifically, the timing of assembly operations can be further obtained based on location and category, thereby determining the first, second, and third stages.

[0081] As an example, the preparation phase is defined as the hand moving towards and gradually approaching a component or tool without making contact. The execution phase is defined as the hand making contact with the component or tool, or the component making contact with the assembly platform. The operation is defined as the completion phase when the hand leaves the object or the object is stably placed on the platform.

[0082] Step S4024: Determine the action sequence based on the first stage, the second stage, and the third stage.

[0083] After determining the first, second, and third stages, they can be integrated to obtain the action sequence.

[0084] Specifically, step S4024 includes: Step a1: Based on the first stage, the second stage, and the third stage, determine the spatial relationships corresponding to the associated information; wherein, the spatial relationships include at least one of the following: intersection-union ratio, Euclidean distance, relative position, and motion trajectory.

[0085] Spatial relationships can be understood as the relationships between the positions, distances, movement trends, and degree of overlap of various targets (hands, parts, tools, assembly platforms) in the associated information.

[0086] The intersection-union ratio (IUU) can be understood as a measure of the degree of overlap between two target areas (such as a hand and a part, or a part and a platform). It is calculated as the ratio of the intersection area to the union area of ​​the two target areas, with a value range of 0-1. The larger the value, the higher the degree of overlap.

[0087] Euclidean distance can be understood as a measure of the straight-line distance between pixels of two targets (such as a hand and a part, or a tool and a part) in an image coordinate system, intuitively reflecting the distance between the targets.

[0088] Relative position can be understood as the orientation (such as up, down, left, right, center) and offset of another target (such as a part) relative to a certain target (such as an assembly platform).

[0089] Motion trajectory can be understood as the path of position change of the same target (such as a hand or part) in consecutive frame images in chronological order, and is used to determine the target's direction of motion and motion state.

[0090] Specifically, spatial relationship features are extracted by analyzing the position, distance, and motion changes of each target within the three stages.

[0091] As an example, by combining the temporal characteristics of the first, second, and third stages, for the hands, parts, tools, and assembly platforms in the associated information, at least one spatial relationship among intersection-union ratio, Euclidean distance, relative position, and motion trajectory is calculated and determined. Through multi-dimensional spatial features, the interaction state and motion law of each target in the assembly process are characterized.

[0092] Step a2: Based on spatial relationships, determine the actions to be performed according to preset rules to obtain an action sequence; wherein the preset rules include: In the first stage, hand information indicates that the hand is approaching the part; in the second stage, the intersection ratio of hand information and part information is greater than the intersection ratio threshold; in the third stage, part information indicates that the part moves with the hand, which is determined as a picking action; in the first stage, hand information indicates that the hand carries the part to the assembly platform; in the second stage, the intersection ratio of the part and the platform is greater than the intersection ratio threshold; in the third stage, the hand leaves the part, which is determined as a placing action; in the second stage, the tool or hand moves vertically downward above the part, which is determined as a pressing action; in the second stage, the tool moves reciprocating horizontally, which is determined as a wiping action.

[0093] The specific assembly actions performed by operators during the assembly process can include four types: picking up, placing, pressing, and wiping.

[0094] Specifically, based on the spatial relationships such as intersection-union ratio, Euclidean distance, relative position, and motion trajectory obtained in step a1, and combined with the temporal characteristics of the three stages, the execution action type corresponding to each complete three stages is determined one by one according to the preset action judgment rules. Subsequently, the execution actions corresponding to multiple consecutive three stages are arranged in chronological order, and action tags, timestamps, operation objects, and confidence levels are added to finally form a structured action sequence.

[0095] As an example, in the first stage, the Euclidean distance between the hand and the part gradually decreases, and the relative position changes from separation to proximity; in the second stage, the intersection-to-union ratio of the hand and the part is greater than the intersection-to-union ratio threshold; in the third stage, the movement trajectory of the part completely overlaps with the trajectory of the hand, which is determined to be a picking action, and the action label is marked as picking.

[0096] The assembly operation detection method provided in this embodiment of the invention obtains the location and category information of hand information, component information, tool information and assembly platform information by performing target detection on image sequences. Based on the hand information, component information, tool information and assembly platform information, the assembly operation is divided into three stages, restoring the complete process of the operation and effectively improving the accuracy of action recognition.

[0097] Furthermore, by calculating the intersection-union ratio, Euclidean distance, relative position and motion trajectory between the hand, parts, tools and platform in stages, and accurately determining assembly actions such as picking up, placing, pressing and wiping according to preset rules, the system can standardize actions through temporal and spatial joint reasoning, thereby more effectively improving the accuracy of action recognition.

[0098] In step S303 of this embodiment, determining the process information and process status corresponding to the action sequence based on the action sequence can be achieved using, for example: Figure 5 Steps S5031 to S5032 are implemented.

[0099] Step S5031: Determine process information based on the action sequence and the pre-built assembly process standard model; wherein the pre-built assembly process standard model is used at least to indicate the combination of process actions.

[0100] A standard assembly process model can be understood as a standardized model established in advance based on the product assembly process. It is used to indicate at least the combination of process actions required to complete a certain process, and may also include necessary actions, optional actions, prohibited actions, process sequence dependencies, etc.

[0101] A process action combination can be understood as a set of actions necessary to complete a certain standard assembly process, used to determine which process the current operation belongs to.

[0102] Specifically, a pre-built standard model of assembly process is loaded, and the identified action sequence is compared with the process action combination corresponding to each process in the model. Based on information such as action type, action sequence, and action quantity, one or more processes that are most likely to correspond are initially selected to obtain preliminary process information.

[0103] As an example, the action sequence could be picking up a screw, placing the screw, and tightening it. This action combination matches the screw tightening action in the standard assembly process model, directly identifying the current process as the screw tightening action. The action sequence could also include picking up a gasket and placing it for wiping. Searching the standard assembly process model for processes containing similar action combinations initially filters out candidate processes such as gasket assembly and surface pretreatment. Based on the action sequence's composition, process number OP10 is matched in the standard assembly process model, and its process number and name are directly used as the current process information.

[0104] Step S5032: Based on the action sequence and the process action combination, determine the process matching degree between the action sequence and the process action combination.

[0105] Process matching degree can be used as a quantitative indicator to characterize the degree of similarity or agreement between the identified action sequence and the standard process action combination. It can be expressed by means of score, proportion, probability, etc.

[0106] Specifically, the action type, sequence, quantity, and completion status in the action sequence are compared item by item with the corresponding standard process action combination. Through weighted calculation, similarity calculation, completion statistics, and other methods, a quantitative process matching degree is obtained.

[0107] As an example, a standard action combination for a certain process contains four required actions. If the action sequence matches three of them, the process matching degree is 75%. Required actions are given high weight, optional actions are given low weight, and correct action order earns extra points. The total score is used as the matching degree. The similarity between the action sequence and the standard action combination is calculated using methods such as edit distance and cosine similarity, and normalized to serve as the process matching degree. A temporal matching score is calculated based on the degree to which the order of actions matches the standard process flow, and this score is also used as the process matching degree.

[0108] Step S5033: Based on the process matching degree, determine the process information and process status corresponding to the action sequence.

[0109] Specifically, based on the matching results, the current process is identified, and it is determined whether the process is in a state of not started, in progress, completed, or abnormal.

[0110] As an example, a matching threshold is set. When the matching degree reaches or exceeds the corresponding threshold, the final assigned process information is determined. At the same time, the process status is judged and updated based on the matching degree, the completion status of the action, and whether any violations have occurred.

[0111] Please refer to Figure 6 , Figure 6 This is a schematic diagram of process modeling and mapping provided according to an embodiment of the present invention.

[0112] In one possible implementation, combining Figure 6 As shown, Figure 6 The complete process route for product assembly is shown. Specifically: W1: Place the outer casing (no pre-process dependencies); W2: Remove the protective film (dependent on W1); W3: Install the battery (dependent on W2); W4: Close the inner cover (dependent on W3).

[0113] Each process specifies three types of action rules. Among them, the necessary actions are the set of actions that must be performed to complete the process, such as picking up and placing in W1, picking up and placing in W3, and picking up and pressing in W4.

[0114] Optional actions: Non-essential but executable auxiliary actions, such as wiping W1, which do not affect the determination of process status but can be used for process optimization analysis.

[0115] Prohibited actions: Actions that are not allowed during the execution of a process, such as the tightening of W3. If such actions occur, it will be considered a process abnormality.

[0116] For action sequence caching, the cache stores the identified action sequences in time windows. Each action contains three types of information: action type, timestamp, and operation object. Examples include "take, t1, shell", "place, t2, shell", and "remove, t4, protective film". The cached action sequences are continuously updated as the assembly process progresses. The time-series action sequences in the cache are the direct data source for matching subsequent processes. The system dynamically determines the current process and execution progress by comparing the cached actions with the standard process model in real time through a sliding window.

[0117] For a process state machine, four process states are defined. Process states can include: not started, in progress, completed, and abnormal. Details are as follows: The "Not Started" state indicates that the process has not yet been triggered and no necessary actions for the process have been detected.

[0118] The "in progress" status indicates that the first necessary action of the process has been detected, and the process is in the process of execution.

[0119] The completed state indicates that all necessary actions of the process have been completed and no prohibited actions have occurred.

[0120] Abnormal states can indicate that a prohibited action has been detected, or that a required action has timed out and not been completed.

[0121] The state transition rule can be from the start state to the in-progress state: when the first required action of a process appears in the cached action sequence, the state automatically switches to the in-progress state.

[0122] The transition from the in-progress state to the completed state can be represented as the state switching to the completed state when all necessary actions of the process have been completed and no prohibited actions have occurred.

[0123] The transition from an in-progress state to an abnormal state can be characterized by the state switching to an abnormal state when a prohibited action of the process is detected, or when a necessary action is timed out and not completed.

[0124] The state machine outputs the current state of each process. For example, W1 = incomplete, W2 = completed, W3 = in progress, and W4 = not started.

[0125] The assembly operation detection method provided in this embodiment of the invention can map scattered low-level actions to the process level by matching the identified action sequence with a pre-built assembly process standard model and obtaining the process matching degree, thereby accurately locating the currently executed process content and running status, and thus effectively improving the accuracy of action recognition.

[0126] In step S304 of this embodiment, based on process information and process status, a detection operation is performed to obtain the target detection result, which can be achieved using methods such as... Figure 7 Steps S7041 to S7042 are implemented.

[0127] Step S7041: Based on process information and process status, perform missing assembly detection, incorrect assembly detection, and process sequence detection to obtain the first detection result corresponding to the missing assembly detection, the second detection result corresponding to the incorrect assembly detection, and the third detection result corresponding to the process sequence detection.

[0128] Missing parts detection: Based on the process status and completion status of necessary actions, determine whether there are any necessary processes that have not been performed or key components that have not been assembled. Missing parts detection includes: determining whether any necessary processes have not been performed or have not been completed based on the process status and completion rate of necessary actions.

[0129] Misassembly detection can be understood as determining whether an assembly action is irregular, whether the assembly object is incorrect, or whether the assembly position is out of range. Specifically, misassembly detection includes: determining whether the current action is a prohibited action, whether the object being operated on is a permitted object, and whether the placement position is within a preset area.

[0130] Process sequence detection can be understood as determining whether the current process is executed according to the preset process sequence and whether the preceding dependent processes have been completed. Specifically, process sequence detection involves determining whether all the preceding dependent processes of the current process have been completed.

[0131] The first test result can be understood as the output of the missing item detection, including whether it is missing, not missing, the location and extent of the missing item, etc.

[0132] The second detection result can be understood as the output of the misassembly detection, including whether it is misassembled, not misassembled, the type and location of the misassembly, etc.

[0133] The third inspection result can be understood as the output of the process sequence inspection, including whether the sequence is correct, incorrect, or missing previous processes.

[0134] Specifically, based on determining the current process information and process status, parallel detection is performed from three dimensions: missing parts, incorrect parts, and process sequence, to comprehensively identify abnormalities in the assembly process.

[0135] As an example, based on the process information, the necessary actions and parts for the process can be queried. If the process status is completed, but some parts have not yet been placed or clamped, it is determined that there are missing parts. The first detection result is that there are missing parts.

[0136] As an example, the current action is a prohibited action in the process standard model and is judged as incorrect assembly. The second detection result is that there is a violation action.

[0137] Step S7042: Based on the first detection result, the second detection result, and the third detection result, the target detection result is obtained.

[0138] Specifically, the three types of detection results are comprehensively judged, filtered according to preset priorities, and the final target detection result is output.

[0139] As an example, the system obtains the priority of each of the missing, incorrect, and process sequence inspections, filters the inspection results according to their priority, and selects the abnormal result with the highest priority as the final target inspection result; if there are no abnormalities, the system outputs that the inspection is qualified.

[0140] Please refer to Figure 8 , Figure 8 This is a schematic diagram of a multi-level alarm mechanism method provided according to an embodiment of the present invention.

[0141] In one possible implementation, if the output detection fails, the alarm level can be determined based on the target detection result.

[0142] After performing inspections for missing parts, incorrect parts, and process sequence and obtaining the target inspection results, the system maps the inspection results into three alarm levels—Level 1 prompt, Level 2 warning, and Level 3 blockage—based on the error type and severity through the alarm level determination module. The system then responds according to the preset priority order, thereby achieving hierarchical control of the assembly process.

[0143] Level 1 Alarm: The process execution time is approaching the preset upper limit, posing a risk of timeout; Optional actions are missing (not affecting process completion, but may affect process consistency); Action confidence is at a low level between 0.6 and 0.8.

[0144] Alarm methods include displaying a yellow warning message on the interface and broadcasting a warning message via voice.

[0145] The handling strategy can be to automatically record the prompt information and continue the production process; or not to interrupt production, but only as a preventative reminder, with follow-up by regular quality inspections.

[0146] Example: The process is about to time out, and the system prompts "The current process has completed 80% of the expected time. Please monitor the progress."

[0147] Level 2 alarm: Incorrect process sequence (e.g., subsequent processes are executed before the preceding process is completed); missing items are detected during the missing parts inspection; process execution timeout has occurred but no irreversible impact has been caused.

[0148] Alarm methods include displaying an orange warning message on the interface; flashing the equipment indicator light three times accompanied by a buzzer; and sending alarm data to the manufacturing execution system and recording it synchronously in the production log.

[0149] The handling strategy can be to require manual confirmation of warning messages by operators; if necessary, to perform rework operations and correct the assembly process.

[0150] Example: If the protective film is not removed, the system will warn "Protective film detected, please confirm before continuing".

[0151] Level 3 alarm: Misassembly detected as a prohibited action (such as an operation explicitly prohibited in the process standard model); serious quality problem (a clear error with a detection confidence level > 0.85, such as key parts installed backwards or the model not matching); multiple consecutive Level 2 warnings have not been addressed, indicating a significant quality risk.

[0152] The alarm methods can be a full-screen red blocking prompt or a continuous, rapid alarm sound. The handling strategy can be that operators must manually intervene to confirm the error; only after troubleshooting and confirming that there is no error can the blockage be lifted and production resumed.

[0153] Example: If the FPC cable is installed incorrectly, the system will trigger a shutdown and display the message "Cable direction is incorrect, production line is suspended".

[0154] Specifically, step S7042 includes: Step b1: Obtain the first priority corresponding to the first detection result, the second priority corresponding to the second detection result, and the third priority corresponding to the third detection result; wherein, the second priority is greater than the third priority, and the third priority is greater than the first priority.

[0155] The first priority can be understood as the processing priority assigned to the results of missing parts detection, which is the lowest priority in this scheme. The second priority can be understood as the processing priority assigned to the results of incorrect parts detection, which is the highest priority in this scheme. The third priority can be understood as the processing priority assigned to the results of process sequence detection, which is the middle priority in this scheme.

[0156] Specifically, priority rules for test results are pre-configured, specifying that the priority of misassembly test results (second test results) is greater than the priority of process sequence test results (third test results), and the priority of process sequence test results (third test results) is greater than the priority of missing test results (first test results). After obtaining the three types of test results (missing, misassembly, and process sequence), their corresponding first priority, second priority, and third priority are read respectively to form a mapping relationship between priority and test results.

[0157] Step b2: Based on the first priority, second priority, and third priority, the first detection result, the second detection result, and the third detection result are screened to determine the detection result with the highest priority, and this detection result is used as the target detection result.

[0158] The three types of detection results are associated with their corresponding priorities. Then, the detection results are compared and filtered in descending order of priority. If a high-priority detection result is abnormal, it is directly used as the target detection result. If a high-priority result is normal, the next highest priority result is compared, and so on, until the detection result with the highest priority is finally output.

[0159] In one possible implementation scenario, at a smartwatch assembly station, the system has already performed three types of detection: missing parts, incorrect parts, and incorrect process sequence. First inspection result (missing parts inspection): All necessary parts have been assembled, and the result is normal; Second inspection result (misinstallation inspection): The battery model does not match the process requirements, and the result is abnormal; Third test result (process sequence test): All preceding processes have been completed, and the result is normal.

[0160] The system reads the priority of three types of inspection results: the second priority (misinstallation) is higher than the third priority (process sequence), and the third priority (process sequence) is higher than the first priority (missing installation). Based on priority comparison and filtering, the misinstallation inspection result is abnormal and has the highest priority. Therefore, the incorrect battery model is selected as the target inspection result, triggering a level-three blocking alarm. This prompts the operator to replace the battery with the correct model, preventing unqualified products from flowing into the next process, thus achieving priority control of high-risk errors.

[0161] The assembly operation detection method provided in this embodiment of the invention, based on the determined process information and process status, simultaneously performs three detections: missing parts, incorrect parts, and process sequence. It can perform comprehensive and multi-dimensional error prevention judgment on the assembly process, fully covering the problems of missing parts, incorrect parts, and incorrect process sequence that are most likely to occur in manual operation, thereby effectively improving the accuracy of action recognition.

[0162] Furthermore, by setting priorities for missing parts detection, incorrect parts detection, and process sequence detection, and filtering results according to priority, the assembly error with the highest risk and greatest impact can be prioritized when multiple anomalies occur, thereby improving assembly safety.

[0163] In some embodiments, the assembly operation detection device of the present invention can be implemented in a combination of hardware and software. As an example, the assembly operation detection device of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the assembly operation detection method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0164] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0165] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the detection method for any of the above-mentioned assembly operations. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the detection method for the assembly operation shown in any embodiment of the present invention by calling the computer program.

[0166] In one alternative embodiment, an electronic device is provided, such as Figure 9 As shown, Figure 9The illustrated electronic device 9000 includes a processor 9001 and a memory 9003. The processor 9001 and the memory 9003 are connected, for example, via a bus 9002. Optionally, the electronic device 9000 may further include a transceiver 9004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 9004 is not limited to one type, and the structure of the electronic device 9000 does not constitute a limitation on the embodiments of the present invention.

[0167] Processor 9001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 9001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0168] Bus 9002 may include a path for transmitting information between the aforementioned components. Bus 9002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 9002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus 9002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0169] The memory 9003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0170] The memory 9003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 9001. The processor 9001 executes the application code stored in the memory 9003 to implement the content shown in the foregoing method embodiments.

[0171] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0172] It should be noted that, Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0173] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a detection method for any of the above-mentioned assembly operations.

[0174] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0175] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the detection method for the assembly operation described above.

[0176] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0177] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0178] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0179] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0180] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0181] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0182] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0183] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting assembly operations, characterized in that, The method includes: Acquire video streams from the assembly work area and determine image sequences based on the video streams; Identify the assembly operations corresponding to the image sequence to obtain the action sequence; Based on the action sequence, determine the process information and process status corresponding to the action sequence; Based on the process information and the process status, a detection operation is performed to obtain the target detection result.

2. The method for detecting assembly operations according to claim 1, characterized in that, The process of identifying the assembly operation corresponding to the image sequence to obtain an action sequence includes: Target detection is performed on the image sequence to obtain the association information of the image sequence; wherein, the association information includes at least one of hand information, component information, tool information, and assembly platform information; Based on the association information, determine the location and category corresponding to the association information; Based on the location and category, the assembly operation is divided into a first stage, a second stage, and a third stage; wherein the first stage is executed before the second stage, and the second stage is executed before the third stage. The action sequence is determined based on the first stage, the second stage, and the third stage.

3. The method for detecting assembly operations according to claim 2, characterized in that, Determining the action sequence based on the first stage, the second stage, and the third stage includes: Based on the first stage, the second stage, and the third stage, the spatial relationship corresponding to the associated information is determined; wherein, the spatial relationship includes at least one of intersection-union ratio, Euclidean distance, relative position, and motion trajectory; Based on the spatial relationship, actions are determined according to preset rules to obtain an action sequence; wherein, the preset rules include: In the first stage, hand information indicates that the hand is approaching the part; in the second stage, the intersection ratio of hand information and part information is greater than the intersection ratio threshold; in the third stage, part information indicates that the part moves with the hand, which is determined as a picking action; in the first stage, hand information indicates that the hand carries the part to the assembly platform; in the second stage, the intersection ratio of the part and the platform is greater than the intersection ratio threshold; in the third stage, the hand leaves the part, which is determined as a placing action; in the second stage, the tool or hand moves vertically downward above the part, which is determined as a pressing action; in the second stage, the tool moves reciprocating horizontally, which is determined as a wiping action.

4. The method for detecting assembly operations according to claim 1, characterized in that, The step of determining the process information and process status corresponding to the action sequence based on the action sequence includes: Based on the action sequence and the pre-built assembly process standard model, process information is determined; wherein, the pre-built assembly process standard model is used at least to indicate the combination of process actions; Based on the action sequence and the process action combination, determine the process matching degree between the action sequence and the process action combination; Based on the process matching degree, the process information and process status corresponding to the action sequence are determined.

5. The method for detecting assembly operations according to claim 1, characterized in that, The step of performing a detection operation based on the process information and the process status to obtain the target detection result includes: Based on the process information and the process status, missing assembly detection, incorrect assembly detection, and process sequence detection are performed to obtain a first detection result corresponding to the missing assembly detection, a second detection result corresponding to the incorrect assembly detection, and a third detection result corresponding to the process sequence detection. Based on the first detection result, the second detection result, and the third detection result, the target detection result is obtained.

6. The method for detecting assembly operations according to claim 5, characterized in that, The step of obtaining the target detection result based on the first detection result, the second detection result, and the third detection result includes: Obtain the first priority corresponding to the first detection result, the second priority corresponding to the second detection result, and the third priority corresponding to the third detection result; wherein, the second priority is greater than the third priority, and the third priority is greater than the first priority; Based on the first priority, the second priority, and the third priority, the first detection result, the second detection result, and the third detection result are filtered to determine the detection result with the highest priority, and this detection result is used as the target detection result.

7. The method for detecting assembly operations according to claim 5, characterized in that, The missing parts detection includes: determining whether any necessary processes have not been performed or completed based on the process status and the completion rate of necessary actions; The misplacement detection includes: determining whether the current action is a prohibited action, whether the object being operated on is a permitted object, and whether the placement position is within a preset area; The process sequence detection is as follows: determine whether all the preceding dependent processes of the current process have been completed.

8. A detection device for assembly operations, characterized in that, The device includes: The acquisition and determination module is used to acquire video streams from the assembly work area and determine image sequences based on the video streams; The recognition module is used to recognize the assembly operation corresponding to the image sequence to obtain the action sequence; The process determination module is used to determine the process information and process status corresponding to the action sequence based on the action sequence; The detection execution module is used to perform detection operations based on the process information and the process status to obtain the target detection result.

9. An electronic device, characterized in that, include: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to cause the processor to perform a detection method for an assembly operation as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the assembly operation of any one of claims 1-7.

Images