A method, apparatus and device for pipeline assembly detection

Abstract

The application discloses a pipeline assembly standard detection method, device and equipment, the method comprises the following steps: acquiring a real-time monitoring picture of a preset assembly area, identifying whether a preset marker bound to a product to be assembled exists in the real-time monitoring picture; when it is determined that the current is in the assembly stage according to the identification result, acquiring a real-time monitoring picture of at least one operation area corresponding to the product to be assembled; performing human body identification on the real-time monitoring picture of the at least one operation area; and determining whether the assembly steps performed conform to the assembly standard according to whether a human body is identified in the operation area. The pipeline assembly standard detection method provided by the application is not limited by the model of the equipment to be assembled and the assembly steps, and can meet the differentiated assembly process.

Description

Technical Field

[0001] This application belongs to the field of video processing technology, and in particular relates to a method, apparatus and equipment for compliance testing of assembly lines. Background Technology

[0002] Assembly line workers often perform multiple repetitive operations during the assembly process. Assembly workers need to retrieve parts from multiple areas and assemble them onto the equipment on the assembly line for a single, compliant operation to be considered complete. Human fatigue can easily lead to non-compliant assembly, affecting product quality and efficiency. Using video-based methods to provide timely warnings of non-compliant operations can help companies improve production quality and efficiency. However, there are many different equipment models from various manufacturers, with varying appearances and installation procedures. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, and equipment for compliance testing of assembly line equipment, which does not limit the model of the equipment to be assembled or the assembly steps, and can meet the needs of differentiated assembly processes.

[0004] Firstly, this application provides a method for compliance inspection of assembly line processes, the method comprising:

[0005] Obtain real-time monitoring footage of the preset assembly area and identify whether there are preset markers in the real-time monitoring footage that are bound to the product to be assembled;

[0006] When the identification results determine that the assembly stage is currently underway, real-time monitoring images of at least one operation area corresponding to the product to be assembled are obtained.

[0007] Human body recognition is performed on the real-time monitoring screen of at least one operating area;

[0008] Whether the assembly steps performed conform to the assembly specifications is determined based on whether a human body is detected in the operating area.

[0009] In one or more embodiments, determining that the current stage is the assembly stage based on the identification result includes:

[0010] Based on the real-time monitoring footage, the assembly is determined to begin when a preset identifier bound to the product to be assembled is first identified in the preset assembly area.

[0011] Once assembly begins, the assembly is considered complete when a preset identifier bound to the product to be assembled is detected outside the preset assembly area, based on the real-time monitoring screen.

[0012] In one or more embodiments, the preset identifier is an identification code used to identify the type of product to be assembled.

[0013] In one or more embodiments, the identification code includes at least one of a QR code, a barcode, and a fixed character.

[0014] In one or more embodiments, determining whether a human body is detected in the operating area is accomplished in the following manner:

[0015] When a human joint is identified in the real-time monitoring screen, it is detected whether the coordinates of the human joint overlap with the coordinate range of the pre-defined operation area in the real-time monitoring screen.

[0016] If the coordinates of the human body joint point overlap with the coordinate range corresponding to the operation area, it is determined that a human body has been identified in the operation area.

[0017] In one or more embodiments, human body recognition is performed on the real-time monitoring screen of the at least one operating area, including:

[0018] Human images in the real-time monitoring footage are detected based on deep learning algorithms;

[0019] Human pose recognition algorithm is used to detect human joints in the human image and obtain the coordinates of the human joints in the real-time monitoring screen.

[0020] In one or more embodiments, determining whether the performed assembly steps conform to assembly specifications based on whether a human body is detected in the operating area includes:

[0021] Obtain all operation areas that have detected the human body, and sort all operation areas according to the time sequence in which the human body was detected;

[0022] Based on the correspondence between the operation areas and the assembly steps, determine the assembly steps and assembly sequence to be executed for all ordered operation areas.

[0023] The assembly steps and sequence corresponding to all the orderly operation areas are matched with the multiple assembly steps and sequences corresponding to the predefined products to be assembled, and the assembly conformity is determined based on the matching results.

[0024] In one or more embodiments, determining whether the performed assembly steps conform to assembly specifications based on whether a human body is detected in the operating area includes:

[0025] If no human body is detected in the operating area, it is determined that the assembly step was not performed and the assembly does not meet the specifications.

[0026] In one or more embodiments, the preset marker is used to identify different types of products to be assembled, and different types of products to be assembled correspond to different assembly steps and assembly sequences.

[0027] Secondly, this application provides an apparatus for compliance testing of assembly lines, the apparatus comprising:

[0028] The first screen acquisition module is used to acquire the real-time monitoring screen of the preset assembly area and identify whether there is a preset identifier in the real-time monitoring screen that is bound to the product to be assembled.

[0029] The second screen acquisition module is used to acquire real-time monitoring screens of at least one operation area corresponding to the product to be assembled when the recognition result determines that the current stage is assembly.

[0030] Human body recognition module, used to perform human body recognition on the real-time monitoring screen of the at least one operating area;

[0031] The assembly compliance inspection module is used to determine whether the assembly steps performed comply with assembly specifications based on whether a human body is detected in the operation area.

[0032] Thirdly, embodiments of this application provide a production line assembly compliance inspection device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a production line assembly compliance inspection method as described in any of the first aspects of this application.

[0033] Fourthly, embodiments of this application also provide a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of a terminal device, enables the terminal device to perform the assembly line compliance inspection method as described in any of the first aspects of this application.

[0034] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0035] This application provides a method, apparatus, and equipment for compliance testing of assembly line products. The method determines the start and end of an assembly process by using a preset identifier bound to the product to be assembled to determine whether it is in the assembly area. During the assembly process, the method determines whether the assembly steps performed comply with the assembly specifications of the equipment to be assembled by whether a human body is detected in the operation area. The method does not limit the model of the equipment to be assembled or the assembly steps, and can meet the needs of differentiated assembly processes. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of an assembly line scenario provided in an embodiment of this application;

[0038] Figure 2 A flowchart of the assembly line compliance testing method provided in this application embodiment;

[0039] Figure 3 This is a schematic diagram illustrating the human joint detection effect provided in an embodiment of this application;

[0040] Figure 4 This is a flowchart of a method for detecting assembly fit based on assembly parts, provided in an embodiment of this application.

[0041] Figure 5 This is a flowchart of a method for detecting assembly fit based on assembly sequence, provided in an embodiment of this application.

[0042] Figure 6 An overall flowchart illustrating the compliance of the testing assembly line provided in this application embodiment;

[0043] Figure 7 A schematic diagram of an assembly line compliance testing device provided in an embodiment of this application;

[0044] Figure 8 A schematic diagram of the equipment for compliance testing of assembly line assembly provided in this application embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0046] Assembly line workers often perform multiple repetitive operations during assembly. An assembly worker needs to retrieve parts from multiple areas and assemble them onto the equipment on the assembly line for a single, compliant operation to be considered complete. Human fatigue can easily lead to non-compliant assembly, impacting product quality and efficiency. Using video-based methods to provide timely warnings of non-compliant operations can help companies improve production quality and efficiency. However, there are many equipment models from different manufacturers, with varying appearances and installation procedures. Deep learning-based models need to be retrained for different manufacturers' assembly videos, lacking universality.

[0047] In view of the above problems, this application provides a method, apparatus and equipment for compliance inspection of assembly line. The method uses a preset identifier bound to the product to be assembled to determine the start and end of an assembly by whether it is in the assembly area. During the assembly process, the method determines whether the assembly steps performed comply with the assembly specifications of the equipment to be assembled by whether a human body is detected in the operation area. The method does not limit the model of the equipment to be assembled or the assembly steps, and can meet the needs of differentiated assembly processes.

[0048] like Figure 1 The diagram shown is a schematic of a production line assembly scenario provided in an embodiment of this application. When a device 101 to be assembled moves to the current assembly area, it indicates that the device 101 to be assembled has begun to be assembled, and the device 101 to be assembled needs to be fitted with the corresponding parts.

[0049] For example, the device 101 to be assembled needs to assemble two parts, including part 102 and part 103. (Refer to...) Figure 1 Parts 102 and 103 each correspond to an operating area. Workers need to go to these two operating areas to retrieve the parts and assemble them onto the assembly equipment 101. Once the workers have assembled all the parts, the assembly equipment 101 will move out of the assembly area, indicating that the assembly equipment 101 has been completed.

[0050] based on Figure 1 The assembly scenario shown is as follows: Figure 2 The diagram shown is a flowchart of a compliance testing method for assembly lines provided in this application, which includes the following steps:

[0051] S201, Obtain the real-time monitoring screen of the preset assembly area, and identify whether there is a preset identifier in the real-time monitoring screen that is bound to the product to be assembled;

[0052] The aforementioned preset identifier bound to the product to be assembled is an identification code used to identify the type of the product to be assembled. The identification code includes at least one of a QR code, barcode, or fixed characters. In specific implementations, the preset identifier can also be other identifiers that can be used to identify the type of product to be assembled; this application does not limit this. The preset identifier is used to identify different types of products to be assembled, and different types of products to be assembled correspond to different assembly steps and assembly sequences.

[0053] For example, when the preset identifier is a QR code, different QR codes correspond to different types of products to be assembled, and different assembly steps or different assembly steps and sequences. On the one hand, when the QR code only corresponds to different assembly steps, it indicates that the current product to be assembled has no requirement on the order in which the parts to be assembled are placed. On the other hand, when the QR code corresponds to different assembly steps and sequences, it indicates that the current product to be assembled has a requirement on the order in which the parts to be assembled are placed.

[0054] If a QR code is detected in the preset assembly area in the real-time monitoring screen, it indicates that the product of a certain type corresponding to the QR code has entered the assembly area, and an assembly begins.

[0055] S202, when it is determined from the identification result that the current stage is assembly, obtain the real-time monitoring screen of at least one operation area corresponding to the product to be assembled;

[0056] As a feasible implementation method, the embodiments of this application predefine an assembly area and at least one operation area corresponding to at least one part to be assembled. The assembly area is the area where workers assemble the products to be assembled on the assembly line, while the operation area corresponds to the area where workers pick up the corresponding parts to be assembled. The assembly area and each operation area are independent of each other.

[0057] S203, perform human body recognition on the real-time monitoring screen of the at least one operating area;

[0058] Once an assembly begins, corresponding parts need to be assembled onto the product to be assembled. These parts are placed in different operating areas, so workers need to go to these operating areas to retrieve these parts. Correspondingly, whether a human body is detected in the operating area can determine whether the worker has retrieved the part corresponding to that operating area. Therefore, human body recognition needs to be performed on the real-time monitoring screen of at least one operating area during the assembly process. The human body recognition process includes:

[0059] The system detects human images in the real-time monitoring screen using a deep learning algorithm, and detects human joints in the human images using a human pose recognition algorithm, and obtains the coordinates of the human joints in the real-time monitoring screen.

[0060] After obtaining the coordinates of the human body's joint points in the real-time monitoring screen using the above method, in one or more possible embodiments, it can be determined that a human body has been identified in the operation area based on whether the coordinates of the human body's joint points in the real-time monitoring screen overlap with the coordinate range corresponding to a pre-defined operation area in the real-time monitoring screen. In one or more possible embodiments, if the coordinates of the joint point corresponding to the human body's hand overlap with the coordinate range corresponding to the pre-defined operation area in the real-time monitoring screen, it is considered that the part corresponding to that operation area has been removed.

[0061] As a feasible implementation method, when performing human detection on real-time monitoring footage using deep learning algorithms, publicly available algorithms such as SSD and YOLO can be used for human detection. Subsequently, based on the human detection performed using deep learning algorithms, publicly available algorithms such as OpenPose can be used to detect human joints in the acquired human images and obtain the coordinates of the human torso and limbs on the image, such as... Figure 3 The diagram shown illustrates the human joint detection effect provided in this embodiment. After obtaining the coordinates of the human torso and limbs on the image, by detecting whether the coordinates of the human joints reach the coordinate range of the operation area corresponding to the target assembly action, it can be determined whether the human body is detected in the real-time monitoring screen of the operation area.

[0062] S204. Determine whether the assembly steps performed conform to the assembly specifications based on whether a human body is detected in the operating area.

[0063] As a feasible implementation method, during the assembly process, once all parts are assembled, the system only checks whether a human body is detected in the pre-defined operating area to determine if the assembly is compliant. In other words, it does not focus on the worker's assembly sequence, but only on whether the assembly steps are complete. For example... Figure 4 The diagram shown is a flowchart of a method for detecting assembly fit based on assembly parts according to an embodiment of this application, including:

[0064] S401, acquire all operation areas of the human body that have been identified;

[0065] S402, Based on the correspondence between the operation area and the assembly step, determine the assembly steps to be executed for all operation areas;

[0066] Each assembly step corresponds to an operation area, and the detection of a human body in each operation area indicates that the assembly step corresponding to that operation area has been executed.

[0067] S403 matches the assembly steps performed in all operation areas with multiple predefined assembly steps corresponding to the product to be assembled, and determines whether the assembly conforms to the specifications based on the matching results.

[0068] The assembly steps that have been confirmed to be executed during the assembly process will be matched with all the predefined assembly steps corresponding to the product to be assembled. If all match, it means that the assembly steps corresponding to the product to be assembled have been executed and all the parts that should be assembled have been assembled. In other words, the assembly meets the specifications.

[0069] As another feasible implementation method, when all parts need to be assembled step by step during the assembly process, it is necessary to identify human bodies in all operating areas and the time sequence of human body identification in each operating area needs to conform to the assembly sequence.

[0070] like Figure 5 The diagram shown is a flowchart of a method for detecting assembly fit based on assembly sequence according to an embodiment of this application, including:

[0071] S501, acquire all operation areas that have identified the human body, and sort all operation areas according to the time sequence of human body identification;

[0072] S502, Based on the correspondence between the operation area and the assembly steps, determine the assembly steps and assembly sequence to be executed for all ordered operation areas.

[0073] Each assembly step corresponds to an operation area, and there is a time sequence in which the human body is detected among the operation areas. Therefore, all the ordered operation areas correspond to the assembly steps and assembly sequence to be executed.

[0074] S503 matches the assembly steps and assembly sequence corresponding to all ordered operation areas with multiple predefined assembly steps and assembly sequences corresponding to the product to be assembled, and determines whether the assembly conforms to the specifications based on the matching results.

[0075] If, during the completed assembly process, the assembly steps and sequence executed in all the orderly operation areas match the predefined assembly steps and sequence corresponding to the product to be assembled, it means that the assembly steps corresponding to the product to be assembled have been executed in sequence, and all the parts that should be assembled have been assembled in sequence. In other words, the assembly is successful. If the matching fails, it means that the assembly has failed, and an alarm needs to be output to remind the relevant personnel.

[0076] It should be noted that the above Figure 4 and Figure 5 The assembly detection process shown occurs after assembly is complete, specifically after a preset marker is detected outside the assembly area. If no human body is detected in the operating area during the completed assembly process, it is determined that an assembly step was not performed and the assembly does not meet specifications. In this case, an alarm should be triggered to alert relevant personnel.

[0077] The following, in conjunction with the foregoing embodiments, provides an overall flowchart for inspecting the compliance of the assembly line, as follows: Figure 6 As shown, the overall process for inspecting assembly line compliance includes three main parts: 610 identification mark inspection, 620 human body inspection, and 630 assembly compliance.

[0078] Step 610, Identifier detection, indicates the start and end of the detection assembly, proceed to step 611;

[0079] Step 611: Determine whether the marker has entered the assembly area. If yes, proceed to step 612; otherwise, proceed to step 613.

[0080] Step 612: The marker enters the assembly area, indicating that assembly has begun. Then proceed to steps 614 and 620.

[0081] Step 613: No marker was detected entering the assembly area, indicating that assembly has not started. Proceed to step 610.

[0082] Step 614: Determine whether the marker has left the assembly area. If yes, proceed to step 615; otherwise, proceed to step 616.

[0083] Step 615: The marker leaves the assembly area, indicating that the assembly is complete. Proceed to step 630.

[0084] Step 616: If no marker is detected leaving the assembly area, it indicates that the assembly is not finished. Proceed to step 612.

[0085] Step 620, Human body detection, indicates that the assembly has started. The system begins to detect human body images in the real-time monitoring screen and executes step 621.

[0086] Step 621, Human Joint Detection, indicates that human joint detection is performed on the human image in the real-time monitoring screen and step 622 is executed.

[0087] Step 622: Determine whether the limb has entered region 1, ..., n, and then proceed to step 623;

[0088] Step 623: Output the judgment result and execute step 614;

[0089] It should be noted that the above judgment results refer to whether any limbs have entered each preset operating area, and whether the worker has taken the parts to be assembled in each operating area and completed the corresponding assembly steps.

[0090] Step 630: Determine if the assembly is compliant. If yes, proceed to step 631; otherwise, proceed to step 632.

[0091] The criteria for determining whether the assembly is compliant are based on the judgment result of step 622 after the assembly is completed in step 615. Figure 4 or Figure 5 The process shown is the same, and will not be described in detail here.

[0092] Step 631: Do not output an alarm;

[0093] Step 632, output alarm. The assembly line compliance detection method provided in this application embodiment uses the presence or absence of a preset identifier bound to the product to be assembled in the assembly area to determine the start and end of an assembly cycle. During the assembly process, it determines whether the executed assembly steps conform to the assembly specifications of the equipment to be assembled based on whether a human body is detected in the operating area. It does not limit the model of the equipment to be assembled or the assembly steps, and has strong versatility, capable of meeting diverse assembly processes. Based on the same inventive concept, this application also provides an assembly line compliance detection device, such as... Figure 7 As shown, it includes:

[0094] The first screen acquisition module 701 is used to acquire the real-time monitoring screen of the preset assembly area and identify whether there is a preset identifier in the real-time monitoring screen that is bound to the product to be assembled.

[0095] The preset identifier is an identification code used to identify the type of product to be assembled, and the identification code includes at least one of QR code, barcode, and fixed characters.

[0096] The preset markers are used to identify different types of products to be assembled, and different types of products to be assembled correspond to different assembly steps and assembly sequences.

[0097] The second screen acquisition module 702 is used to acquire real-time monitoring screens of at least one operation area corresponding to the product to be assembled when the recognition result determines that the current stage is assembly.

[0098] The second screen acquisition module 702 is also used to determine the start of assembly when a preset identifier bound to the product to be assembled is first identified in the preset assembly area based on the real-time monitoring screen.

[0099] Once assembly begins, the assembly is considered complete when a preset identifier bound to the product to be assembled is detected outside the preset assembly area, based on the real-time monitoring screen.

[0100] Human body recognition module 703 is used to perform human body recognition on the real-time monitoring screen of the at least one operating area;

[0101] The human body recognition module 703 is further configured to determine whether a human body is recognized in the operating area using the following methods:

[0102] When a human joint is identified in the real-time monitoring screen, it is detected whether the coordinates of the human joint overlap with the coordinate range of the pre-defined operation area in the real-time monitoring screen.

[0103] If the coordinates of the human body joint point overlap with the coordinate range corresponding to the operation area, it is determined that a human body has been identified in the operation area.

[0104] The process of performing human body recognition on the real-time monitoring screen of at least one operating area includes:

[0105] Human images in the real-time monitoring footage are detected based on deep learning algorithms;

[0106] Human pose recognition algorithm is used to detect human joints in the human image and obtain the coordinates of the human joints in the real-time monitoring screen.

[0107] The assembly compliance detection module 704 is used to determine whether the assembly steps performed comply with the assembly specifications based on whether a human body is detected in the operation area.

[0108] The assembly compliance detection module 704 is also used to acquire all operation areas that have been identified as human bodies, and sort all operation areas according to the time sequence in which the human bodies were identified.

[0109] Based on the correspondence between the operation areas and the assembly steps, determine the assembly steps and assembly sequence to be executed for all ordered operation areas.

[0110] The assembly steps and sequence corresponding to all the orderly operation areas are matched with the multiple assembly steps and sequences corresponding to the predefined products to be assembled, and the assembly conformity is determined based on the matching results.

[0111] The assembly compliance detection module 704 is also used to determine that the assembly step has not been performed and the assembly does not meet the specifications if no human body is detected in the operation area.

[0112] The specific implementation method of the above-mentioned assembly line compliance testing device is described in the foregoing embodiments and will not be detailed here.

[0113] Based on the same inventive concept, this application also provides a compliance testing device 800 for assembly lines, such as... Figure 8 As shown, it includes at least one processor 802; and a memory 801 communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for assembly line compliance inspection.

[0114] The memory 801 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions. The memory 801 may be volatile memory, such as random-access memory (RAM); it may also be non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may be any one or a combination of the above-mentioned volatile and non-volatile memory types.

[0115] The processor 802 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. It can also be a hardware chip. This hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0116] Based on the same inventive concept, embodiments of this application provide a computer program medium, wherein the computer storage medium stores a computer program, and the computer program is used to cause a computer to execute the above-described method for compliance testing of assembly line.

[0117] The aforementioned storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0118] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0119] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0120] The technical solutions provided in this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0121] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood by those skilled in the art from the computer program instructions that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of one or more computer-usable storage media containing computer-usable program code (implementing each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams). These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of inline assembly inspection, the method comprising: The method comprises the following steps: acquiring a real-time monitoring picture of a preset assembly area, and determining that a type of product to be assembled corresponding to a preset marker bound to the product to be assembled has entered the preset assembly area if the preset marker is recognized in the real-time monitoring picture; acquiring a real-time monitoring picture of at least one operation area corresponding to the product to be assembled; the preset assembly area and the at least one operation area are independent of each other; the preset assembly area is used for product assembly of the product to be assembled, and the operation area is used for placing parts to be assembled, and each operation area corresponds to a predefined assembly step of the product to be assembled; performing human body recognition on the real-time monitoring picture of any one of the operation areas, and determining that a part corresponding to any one of the operation areas has been taken if coordinates of a joint corresponding to a hand of a human body recognized in the real-time monitoring picture of at least one of the operation areas overlap with a coordinate range predefined in the real-time monitoring picture; after detecting that the preset marker appears outside the assembly area, acquiring all operation areas in which a human body is recognized, sorting the all operation areas according to a time sequence in which the human body is recognized, determining assembly steps and assembly sequences corresponding to the sorted all operation areas according to a correspondence relationship between the operation areas and the assembly steps, and determining that the assembly is in conformity with a specification if the assembly steps and the assembly sequences corresponding to the sorted all operation areas match a plurality of assembly steps and assembly sequences predefined for the product to be assembled successfully in a completed assembly process.

2. The method of claim 1, wherein, determining a current assembly stage according to a recognition result, comprising: determining that assembly starts when a preset marker bound to a product to be assembled is recognized for the first time in a preset assembly area according to the real-time monitoring picture; determining that assembly ends when the preset marker bound to the product to be assembled is recognized outside the preset assembly area according to the real-time monitoring picture after the assembly starts.

3. The method of claim 1, wherein, The preset marker is an identification code used for identifying a type of product to be assembled.

4. The method of claim 3, wherein, The identification code comprises at least one of a two-dimensional code, a bar code, and fixed characters.

5. The method of claim 1, wherein, The human body recognition on the real-time monitoring picture of any one of the operation areas comprises: detecting a human body image in the real-time monitoring picture based on a deep learning algorithm; detecting a human joint in the human body image based on a human body posture recognition algorithm, and acquiring coordinates of the human joint in the real-time monitoring picture.

6. The method of claim 1, wherein, The method further comprises: determining that an assembly step is not performed and the assembly is not in conformity with a specification if a human body is not recognized in an operation area.

7. The method of claim 1, wherein, The preset marker is used for identifying different types of products to be assembled, and different types of products to be assembled correspond to different assembly steps and assembly sequences.

8. A pipelined gage assembly inspection apparatus, characterized by, The method comprises the following steps: a first picture acquisition module is configured to acquire a real-time monitoring picture of a preset assembly area, and determine that a type of product to be assembled corresponding to a preset marker bound to the product to be assembled has entered the preset assembly area if the preset marker is recognized in the real-time monitoring picture; A second picture acquisition module is configured to acquire real-time monitoring pictures of at least one operation area corresponding to the product to be assembled; wherein the preset assembly area and the at least one operation area are independent of each other; the preset assembly area is used for product assembly of the product to be assembled, and the operation area is used for placing parts to be assembled, and each operation area corresponds to a predefined assembly step of the product to be assembled; A human body recognition module is configured to perform human body recognition on the real-time monitoring pictures of any one operation area, and if the coordinates of the joint points corresponding to the hands of the human body overlap with a predefined coordinate range in the real-time monitoring pictures of the at least one operation area, it is determined that the parts corresponding to any one operation area have been taken. An assembly criterion detection module is configured to, after detecting that the preset marker appears outside the assembly area, acquire all operation areas in which the human body is recognized, sort the all operation areas according to the time sequence in which the human body is recognized, determine the assembly steps and assembly sequences corresponding to the sorted all operation areas according to the correspondence between the operation areas and the assembly steps, and if the assembly steps and assembly sequences corresponding to the sorted all operation areas match the predefined multiple assembly steps and assembly sequences of the product to be assembled in the completed assembly process, it is determined that the assembly conforms to the criterion.

9. A pipelined assembly gauge inspection apparatus, characterized by, The apparatus includes at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

Images