Product assembly method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of AR equipment, and provides a product assembly method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: displaying assembly operation information corresponding to a current assembly process; the assembly operation information is used for guiding a correct operation mode of the current assembly process; based on a user operation instruction for the current assembly process, an assembly equipment operation instruction is sent to target assembly equipment; the assembly equipment operation instruction is used for indicating target assembly equipment to execute corresponding assembly operation; receiving an assembly quality detection result sent by the server; the assembly quality detection result is obtained by performing assembly quality detection on the assembly operation real-time image of the current assembly process by the server; displaying prompt information corresponding to the assembly quality detection result; the prompt information is used for prompting to enter the next assembly process or prompting to improve the current assembly process of which the assembly operation is completed. The product assembling accuracy can be improved, and the product percent of pass is finally improved.

Description

Technical Field

[0001] The present invention relates to the field of AR device technology, and in particular to a product assembly method, device, electronic device, storage medium and program product. Background Art

[0002] Product assembly refers to the process of assembling parts according to the specified technical requirements and making them qualified products after debugging and inspection. For example, the line assembly of electrical equipment requires wiring assembly work.

[0003] At present, product assembly often needs to rely on experienced assemblers. However, the complexity and precision of the products to be assembled are constantly increasing, and the assembly work requires more and more skills from the assemblers. Since experienced assemblers may also make assembly errors, which may lead to product problems, over-reliance on experienced assemblers will lead to a decrease in product qualification rate, that is, low product assembly accuracy. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention proposes a product assembly method, which can know the correct operation mode of the current assembly process with the help of AR equipment, without the need for experienced assemblers, and only needs to display assembly operation information to guide assemblers to operate, thereby reducing assembly errors, improving product assembly accuracy, and thus improving product qualification rate; and there is no need for users to operate the target assembly equipment by themselves, so there is no need to rely on experienced assemblers, thereby improving product assembly accuracy, and ultimately improving product qualification rate; and in the case of assembly errors, reassembly can be performed, thereby improving product assembly accuracy, and ultimately improving product qualification rate, and ensuring that inexperienced assemblers can also know whether they can enter the next assembly process, thereby improving product assembly efficiency.

[0005] The invention also provides a product assembly device.

[0006] The invention also provides an electronic device.

[0007] The present invention also provides a storage medium.

[0008] The present invention also provides a computer program product.

[0009] The product assembly method according to the first aspect of the present invention is applied to an AR device, and the method includes: Displaying assembly operation information corresponding to the current assembly process; the assembly operation information is used to guide the correct operation method of the current assembly process; Based on the user operation instruction for the current assembly process, sending an assembly equipment operation instruction to the target assembly equipment; the assembly equipment operation instruction is used to instruct the target assembly equipment to perform a corresponding assembly operation; receiving an assembly quality inspection result sent by a server; the assembly quality inspection result is obtained by the server performing assembly quality inspection on a real-time image of an assembly operation of the current assembly process; The prompt information corresponding to the assembly quality inspection result is displayed; the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process after the assembly operation has been completed.

[0010] According to the product assembly method of the embodiment of the present invention, assembly operation information corresponding to the current assembly process is displayed, and the assembly operation information is used to guide the correct operation method of the current assembly process, so that the assembler can know the correct operation method of the current assembly process with the help of the AR device. The assembler does not need to be experienced, and only needs to display the assembly operation information to guide the assembler to operate, thereby reducing assembly errors, improving product assembly accuracy, and thus improving product qualification rate; after displaying the assembly operation information corresponding to the current assembly process, based on the user operation instruction for the current assembly process, the assembly device operation instruction is sent to the target assembly device, that is, the user can trigger the user operation instruction of the AR device based on the assembly operation information, so that the AR device sends the assembly device operation instruction to the target assembly device, without the user operating the target assembly device by himself, so there is no need to rely on experienced assemblers, thereby improving product assembly accuracy, Ultimately, the product qualification rate is improved; after the assembly operation information corresponding to the current assembly process is displayed, and the user performs the assembly operation of the current assembly process based on the assembly operation information, the assembly quality inspection result sent by the server is received, and the prompt information corresponding to the assembly quality inspection result is displayed, and the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process, and the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process that has been completed, so that in the case of assembly errors, re-assembly can be performed, thereby improving the accuracy of product assembly, and ultimately improving the product qualification rate, and ensuring that inexperienced assemblers can also know whether they can enter the next assembly process, thereby improving product assembly efficiency, and the assembly quality inspection by the server can reduce the computing resources occupied by the AR device, improve the performance of the AR device, and ultimately improve the product assembly quality.

[0011] According to an embodiment of the present invention, when the assembly quality inspection result is a failure, the receiving the assembly quality inspection result sent by the server includes: Receiving the assembly quality inspection result sent by the server, and prompt information corresponding to the assembly quality inspection result; the prompt information includes guidance information of the current assembly process; Accordingly, the prompt information corresponding to the assembly quality inspection result is displayed, including: The guidance information is displayed in a manner of spatially mapping the guidance information onto a real environment.

[0012] According to an embodiment of the present invention, before receiving the assembly quality inspection result sent by the server, the method further includes: Collecting a real-time image of the assembly operation of the current assembly process; The real-time image of the assembly operation is sent to the server.

[0013] According to an embodiment of the present invention, the assembly operation information is displayed based on a UI interface displayed by the AR device; After sending the assembly equipment operation instruction to the target assembly equipment based on the user operation instruction for the current assembly process, the method further includes: Based on the user operation instruction, update the UI interface in real time; receiving an execution result corresponding to the assembly device operation instruction returned by the target assembly device; Based on the execution result, the UI interface is updated again.

[0014] According to one embodiment of the present invention, the method further includes: Sending the screen image of the AR device to the server, so that the server synchronizes the screen image to the sharing terminal; The shared terminal includes an AR device and / or a user terminal supporting remote video calls.

[0015] According to an embodiment of the present invention, before displaying the assembly operation information corresponding to the current assembly process, the method further includes: Collecting first image information for identifying a user; Sending the first image information to the server; the first image information is used to instruct the server to perform authentication; When the authentication is successful, the assembly operation information sent by the server is received.

[0016] According to an embodiment of the present invention, before displaying the assembly operation information corresponding to the current assembly process, the method further includes: collecting second image information for identifying the product to be assembled; The second image information is sent to the server; the second image information is used to instruct the server to determine the current product to be assembled.

[0017] According to one embodiment of the present invention, the assembly operation information includes a tutorial animation of the current assembly process and assembly operation key information of the current assembly process; the tutorial animation is used to demonstrate the correct operation method of the current assembly process, and the assembly operation key information is used to prompt precautions for the assembly operation.

[0018] A product assembly method according to an embodiment of the second aspect of the present invention is applied to a server, and the method includes: Performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain the assembly quality inspection result; Sending the assembly quality inspection result to the AR device; the assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or prompt to improve the current assembly process that has been completed; Among them, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to the target assembly device based on the user operation instructions for the current assembly process, and the assembly device operation instructions are used to instruct the target assembly device to perform corresponding assembly operations.

[0019] According to the product assembly method of an embodiment of the present invention, the AR device displays assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process, so that the assembler can learn the correct operation method of the current assembly process with the help of the AR device. The assembler does not need to be experienced, and only needs to display the assembly operation information to guide the assembler to operate, thereby reducing assembly errors, improving product assembly accuracy, and further improving product qualification rate; after displaying the assembly operation information corresponding to the current assembly process, the AR device sends assembly device operation instructions to the target assembly device based on the user operation instructions for the current assembly process, that is, the user can trigger the user operation instructions of the AR device based on the assembly operation information, so that the AR device sends the assembly device operation instructions to the target assembly device, and the user does not need to operate the target assembly device by himself, thereby Without relying on experienced assemblers, the accuracy of product assembly is improved, and the product qualification rate is ultimately improved. The server performs assembly quality inspection on the real-time image of the assembly operation of the current assembly process, obtains the assembly quality inspection result, and sends the assembly quality inspection result to the AR device, so that the AR device displays the prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process that has been completed. In the case of assembly errors, re-assembly can be performed, thereby improving the accuracy of product assembly and ultimately improving the product qualification rate. It also ensures that inexperienced assemblers can also know whether they can enter the next assembly process, thereby improving product assembly efficiency. The assembly quality inspection by the server can reduce the computing resources occupied by the AR device, improve the performance of the AR device, and ultimately improve the product assembly quality.

[0020] According to an embodiment of the present invention, the step of performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain the assembly quality inspection result includes: Determine the target object that needs to be detected for the product to be assembled in the current assembly process; the target object is the object of the assembly operation required in the current assembly process; Performing target detection on the real-time image of the assembly operation based on the target object to obtain a target detection result for the target object; An assembly quality inspection is performed based on the target inspection result to obtain an assembly quality inspection result.

[0021] According to one embodiment of the present invention, the product to be assembled is determined based on the following method: receiving second image information sent by the AR device; the second image information is used to identify the product to be assembled, and the second image information is acquired by the AR device; The product to be assembled is determined based on the second image information.

[0022] According to one embodiment of the present invention, before performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain the assembly quality inspection result, the method further includes: Acquire a configured assembly operation information set for a product to be assembled; the assembly operation information set includes information corresponding to the product to be assembled under multiple assembly processes; The assembly operation information corresponding to the current assembly process is sent to the AR device.

[0023] According to an embodiment of the present invention, sending the assembly operation information corresponding to the current assembly process to the AR device includes: Receiving first image information sent by the AR device; the first image information is used to identify a user; It is determined that the authentication based on the first image information is successful, and assembly operation information corresponding to the current assembly process is sent to the AR device.

[0024] According to a third aspect of the present invention, a product assembly device is deployed in an AR device, and the device includes: The first display module is used to display assembly operation information corresponding to the current assembly process; the assembly operation information is used to guide the correct operation method of the current assembly process; An instruction sending module, used for sending an assembly equipment operation instruction to a target assembly equipment based on a user operation instruction for the current assembly process; the assembly equipment operation instruction is used for instructing the target assembly equipment to perform a corresponding assembly operation; A result receiving module is used to receive the assembly quality inspection result sent by the server; the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process; The second display module is used to display prompt information corresponding to the assembly quality detection result; the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process after the assembly operation has been completed.

[0025] According to a fourth aspect of the present invention, a product assembly device is deployed on a server, and the device includes: The quality inspection module is used to perform assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain the assembly quality inspection result; A result sending module is used to send the assembly quality inspection result to the AR device; the assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or prompt to improve the current assembly process that has been completed; Among them, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to the target assembly device based on the user operation instructions for the current assembly process, and the assembly device operation instructions are used to instruct the target assembly device to perform corresponding assembly operations.

[0026] According to an embodiment of the fifth aspect of the present invention, the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, it implements any one of the product assembly methods described above.

[0027] According to the non-transitory computer-readable storage medium of the sixth aspect of the present invention, a computer program is stored thereon, and when the computer program is executed by a processor, the product assembly method as described in any one of the above is implemented.

[0028] According to a seventh aspect of the present invention, a computer program product comprises a computer program, which, when executed by a processor, implements any one of the product assembly methods described above.

[0029] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The assembly operation information corresponding to the current assembly process is displayed, and the assembly operation information is used to guide the correct operation method of the current assembly process, so that the assembler can know the correct operation method of the current assembly process with the help of the AR device. The assembler does not need to be experienced. It only needs to display the assembly operation information to guide the assembler to operate, thereby reducing assembly errors, improving product assembly accuracy, and thus improving product qualification rate; after displaying the assembly operation information corresponding to the current assembly process, based on the user operation instructions for the current assembly process, the assembly equipment operation instructions are sent to the target assembly equipment, that is, the user can trigger the user operation instructions of the AR device based on the assembly operation information, so that the AR device sends the assembly equipment operation instructions to the target assembly equipment, without the user operating the target assembly equipment by themselves, so there is no need to rely on experienced assemblers, thereby improving product assembly accuracy and ultimately improving product qualification rate. rate; after displaying the assembly operation information corresponding to the current assembly process, and the user performing the assembly operation of the current assembly process based on the assembly operation information, receiving the assembly quality inspection result sent by the server, and displaying the prompt information corresponding to the assembly quality inspection result, and the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process, and the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process that has been completed, so that in the case of assembly errors, re-assembly can be performed, thereby improving the accuracy of product assembly, and ultimately improving the product qualification rate, and ensuring that inexperienced assemblers can also know whether they can enter the next assembly process, thereby improving product assembly efficiency, and assembly quality inspection by the server can reduce the computing resources occupied by AR devices, improve the performance of AR devices, and ultimately improve product assembly quality.

[0030] The AR device displays the assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process, so that the assembler can know the correct operation method of the current assembly process with the help of the AR device. The assembler does not need to be experienced. It only needs to display the assembly operation information to guide the assembler to operate, thereby reducing assembly errors, improving product assembly accuracy, and thus improving product qualification rate. After displaying the assembly operation information corresponding to the current assembly process, the AR device sends assembly equipment operation instructions to the target assembly equipment based on the user operation instructions for the current assembly process, that is, the user can trigger the user operation instructions of the AR device based on the assembly operation information, so that the AR device sends the assembly equipment operation instructions to the target assembly equipment, without the user operating the target assembly equipment by themselves, and thus there is no need to rely on experienced people. assemblers, thereby improving product assembly accuracy and ultimately improving product qualification rate; the server performs assembly quality inspection on the real-time image of the assembly operation of the current assembly process, obtains the assembly quality inspection result, and sends the assembly quality inspection result to the AR device, so that the AR device displays the prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process that has been completed, so that in the case of assembly errors, re-assembly can be performed, thereby improving product assembly accuracy and ultimately improving product qualification rate, and ensuring that inexperienced assemblers can also know whether they can enter the next assembly process, thereby improving product assembly efficiency, and the assembly quality inspection by the server can reduce the computing resources occupied by the AR device, improve the performance of the AR device, and ultimately improve the product assembly quality.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is one of the flow charts of the product assembly method provided by the embodiment of the present invention.

[0034] Figure 2 It is a schematic diagram of the UI interface provided by an embodiment of the present invention.

[0035] Figure 3 This is the second flow chart of the product assembly method provided by the embodiment of the present invention.

[0036] Figure 4 This is the third flow chart of the product assembly method provided by the embodiment of the present invention.

[0037] Figure 5 This is the fourth flow chart of the product assembly method provided by the embodiment of the present invention.

[0038] Figure 6 It is one of the structural schematic diagrams of the product assembly device provided by the embodiment of the present invention.

[0039] Figure 7 This is the second structural schematic diagram of the product assembly device provided by the embodiment of the present invention.

[0040] Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradicting each other.

[0043] Product assembly refers to the process of assembling parts according to the specified technical requirements and making them qualified products after debugging and inspection. For example, the line assembly of electrical equipment requires wiring assembly work.

[0044] At present, product assembly often needs to rely on experienced assemblers. For example, as the complexity and precision of electrical equipment continue to increase, wiring work requires increasingly higher skills from assemblers. Even experienced workers may make mistakes in complex wiring environments, leading to equipment failures or safety accidents. Therefore, over-reliance on experienced assemblers will lead to a decrease in product qualification rate, that is, low product assembly accuracy; and training a qualified assembler is often carried out by training the new with the old, which requires a lot of time and labor costs.

[0045] In view of the above problems, the present invention proposes the following embodiments. Figure 1-Figure 5 The product assembly method of the present invention is described.

[0046] Figure 1 is one of the flow charts of the product assembly method provided by the embodiment of the present invention, such as Figure 1 As shown, the product assembly method includes the following steps 110, 120, 130 and 140. The method is applied to an AR (Augmented Reality) device. The AR device can be a head-mounted AR device (such as AR glasses and head-mounted display devices) or a handheld AR device, etc., which is not specifically limited here.

[0047] Step 110: display assembly operation information corresponding to the current assembly process.

[0048] Here, the current assembly process is the process step that needs to be assembled currently. The current assembly process can be an assembly process automatically entered from the previous assembly process, or it can be an assembly process selected by the assembler through the UI (User Interface) interface of the AR device. For ease of understanding, the assembly operation of the embodiment of the present invention is described by taking the wiring operation as an example.

[0049] The assembly operation information is used to guide the correct operation mode of the current assembly process. Based on this, the assembler can learn the correct operation mode of the current assembly process with the help of AR equipment. The assembler does not need to be experienced. The assembly operation information can be displayed to guide the assembler to operate, thereby reducing assembly errors, improving product assembly accuracy, and thus improving product qualification rate.

[0050] In one embodiment, the UI interface displayed based on the AR device displays the assembly operation information corresponding to the current assembly process.

[0051] The assembly operation information may include but is not limited to at least one of the following: a tutorial animation of the current assembly process and assembly operation key information, etc. The tutorial animation is used to show the correct operation method of the current assembly process, and the assembly operation key information is used to prompt the precautions of the assembly operation.

[0052] In one embodiment, the assembly operation information includes a tutorial animation of the current assembly process and assembly operation key information of the current assembly process; the tutorial animation is used to demonstrate the correct operation method of the current assembly process, and the assembly operation key information is used to prompt precautions for the assembly operation.

[0053] In one embodiment, the tutorial animation is determined based on the following method: according to the process route of the product to be assembled, the tutorial animation of each assembly process is designed. For example, the tutorial animation includes animations of parts assembly and standard parts fastening.

[0054] Furthermore, a 3D reconstruction model of the product to be assembled is first built, and then tutorial animations of each assembly process are designed based on the 3D reconstruction model. Furthermore, the 3D reconstruction model is meshed, reduced in number, and retopologically processed to simplify some complex models and improve the efficiency of tutorial animation production.

[0055] For example, the 3D reconstruction model of the product to be assembled is constructed in the following way: using CAD software, specifying materials for each model, modifying parameters, calculating UV coordinates, and adding textures to build a complete 3D model of the product. Furthermore, the 3D reconstruction model is divided into process steps according to the process route to obtain the model corresponding to each assembly process.

[0056] Furthermore, during the tutorial animation creation process, skeletons are created and bound to make the tutorial animation more vivid and lifelike, thereby improving the accuracy of product assembly.

[0057] For example, computer-aided design (CAD) software is used to convert the wiring design of electrical equipment and lines into a high-precision 3D reconstruction model, and a tutorial animation (simulation animation) containing more than 140 embedded process requirements and quality points is produced for display on the mixed reality front end. These tutorial animations show in detail the correct operation method for each wiring step, as well as the quality control points that require special attention. Users can refer to the above tutorial animations in real time while performing wiring operations, which facilitates the learning and training of assembly personnel and ensures the assembly effect, that is, improving the accuracy of product assembly and thus improving the product qualification rate.

[0058] Furthermore, the current assembly process is displayed while the assembly operation information corresponding to the current assembly process is displayed, so that the assembler can browse the current process conveniently.

[0059] Furthermore, while displaying the assembly operation information corresponding to the current assembly process, multiple assembly processes of the product to be assembled are displayed, so that it is convenient for the assembler to browse and switch the current process. Furthermore, the current assembly process (e.g., marked in yellow) among the multiple assembly processes can be distinguished from the other assembly processes in terms of display effect. Furthermore, the completed assembly process (e.g., marked in green) among the multiple assembly processes can be distinguished from the other assembly processes in terms of display effect.

[0060] To achieve human-computer interaction based on AR devices, a human-computer interface interaction can be constructed. For example, the Unity 3D engine is used to develop a human-computer interaction interface to achieve seamless integration with the AR system; assemblers can view and operate these 3D reconstructed models and tutorial animations in a real working environment by wearing AR devices (such as HoloLens devices) to obtain an immersive assembly operation guidance experience.

[0061] For example, a more specific way to develop a human-computer interaction interface using the Unity 3D engine is as follows. First, UI interface design: organize the content to be visualized for each assembly process, design renderings and Unity UI, and the UI interface includes gesture menus, welcome and login interfaces, main pages and auxiliary prompts, etc. Second, SDK (Software Development Kit) interface development: develop unity scripts through the Microsoft HoloLens Mixed Reality Toolkit (Mixed Reality Toolkit, MRTK) to realize data transmission and interaction between AR devices and background services; among them, MRTK is an open source project that provides basic tools and components for developing mixed reality applications. The system mainly uses several functions of MRTK. The user interface is realized through the controls, dialog boxes and buttons provided by MRTK, allowing users to interact with the mixed reality 3D interface. In terms of the input system, MRTK supports gesture recognition, uses gesture events, gesture tracking and hand network functions, generates and uses 3D mesh models of hands, tracks hand positions and gesture states in real time, recognizes gestures such as clicking, picking, manipulating and navigating, and realizes fine interaction. In addition, voice interaction with the graphical interface can be performed by defining and recognizing preset voice commands. The spatial mapping function generates a 3D network of the environment and establishes anchor points in space, so that the graphical interface can be placed in the physical environment and maintain its position. The camera call function uses the MR device camera to collect data to realize the mixed reality interface. Finally, Unity graphic interaction: obtains image data and gestures obtained by the mobile camera, and voice information from the built-in microphone, updates the UI interface in real time after calculation and processing by the background algorithm, and binds the virtual and real three-dimensional coordinates through the Microsoft HoloLens World Coordinate Locking Tool (WLT, World Locking Tools).

[0062] For ease of understanding, the UI interface that displays the assembly operation information corresponding to the current assembly process is as follows: Figure 2 As shown, Figure 2 It includes tutorial animation, assembly operation key points information, current assembly process and multiple assembly processes of the product to be assembled (assembly process 1 to assembly process 21), so that the assembler can browse and switch the current assembly process. Among them, multiple assembly processes can be clicked to switch the current assembly process. Based on this, the electronic standard operating procedure (ESOP) based on augmented reality is realized, that is, Figure 2 It is the interactive interface for electronic work instructions.

[0063] Furthermore, the information of the current assembly process is sent to the server so that the server can know the current assembly process.

[0064] Step 120 : sending assembly equipment operation instructions to target assembly equipment based on the user operation instructions for the current assembly process.

[0065] Specifically, during the assembly operation, the assembler triggers the user operation instruction through the AR device, so that the AR device sends the assembly device operation instruction corresponding to the user operation instruction to the target assembly device.

[0066] Here, the user operation instruction is triggered by the user (assembler) based on the assembly operation information corresponding to the current assembly process. For example, the tutorial animation and assembly operation key information in the assembly operation information intuitively present the position of the parts, assembly sequence and tightening torque requirements, so that the user can operate through the AR device and send assembly equipment operation instructions to the tightening gun (such as starting, stopping and adjusting the tightening torque, etc.).

[0067] In one embodiment, user operation instructions are triggered by gesture recognition. For example, the displayed UI interface supports gesture recognition, uses gesture events, gesture tracking and hand network functions, generates and uses a 3D mesh model of the hand, tracks the hand position and gesture state in real time, recognizes gestures such as clicking, picking up, manipulating and navigating, and realizes fine interaction. That is, the image data and gesture actions are obtained through the camera to determine the corresponding user operation instructions.

[0068] In another embodiment, the user operation instruction is triggered by voice recognition. For example, the voice interaction with the UI interface is performed by defining and recognizing a preset voice command. That is, the voice data is collected by the built-in microphone of the AR device, and the voice data is subjected to voice recognition to determine the user operation instruction.

[0069] The assembly equipment operation instruction is used to instruct the target assembly equipment to perform the corresponding assembly operation. For example, if the assembly equipment operation instruction instructs to adjust the tightening torque and its tightening torque magnitude, the tightening gun performs the tightening operation based on the tightening torque magnitude. Furthermore, the user operation instruction is generated based on the assembly operation information, and the user does not need to determine the specific operation instruction by himself, so there is no need to rely on experienced assembly personnel, thereby improving the accuracy of product assembly and ultimately improving the product qualification rate.

[0070] Here, the target assembly equipment is used to perform assembly operations, that is, equipment that assists assemblers in performing assembly operations. For example, the target assembly equipment is a tightening gun.

[0071] Furthermore, after the above step 120, the UI interface is updated in real time based on the user operation instruction.

[0072] Furthermore, based on the user operation instruction, the operation record corresponding to the user operation instruction is sent to the server, so that the operation record is stored in the server background for subsequent quality control and process optimization analysis, thereby further improving the accuracy of product assembly. Furthermore, the server uploads the operation record to the MES system, thereby realizing a closed-loop process.

[0073] It should be understood that the assembly operation information corresponding to the current assembly process is displayed, and the assembly operation information is used to guide the correct operation method of the current assembly process, so that the user can trigger the user operation instructions of the AR device based on the assembly operation information, and the AR device sends the assembly device operation instructions to the target assembly device. There is no need for the user to operate the target assembly device by himself, and thus there is no need to rely on experienced assembly personnel, thereby improving the accuracy of product assembly and ultimately improving the product qualification rate.

[0074] Step 130: receiving the assembly quality inspection result sent by the server.

[0075] Among them, the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process. Here, the real-time image of the assembly operation can be collected in real time by the AR device and sent to the server, or it can be collected in real time by other image acquisition devices (such as cameras) and sent to the server. That is, the server performs image inspection on the real-time image of the assembly operation to determine the assembly quality inspection result. The specific method of the assembly quality inspection is shown in the following embodiment, which will not be repeated here.

[0076] It should be understood that after the assembly operation information corresponding to the current assembly process is displayed, and the user performs the assembly operation of the current assembly process based on the assembly operation information, the assembly quality inspection results sent by the server are received, so that if the assembly is incorrect, the assembly can be re-performed, thereby improving the product assembly accuracy and ultimately improving the product qualification rate.

[0077] Step 140: display prompt information corresponding to the assembly quality inspection result.

[0078] The prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process after the assembly operation has been completed.

[0079] Specifically, when the assembly quality inspection result is failure, the corresponding prompt information is displayed to prompt improvement of the current assembly process that has been completed; when the assembly quality inspection result is passing, the corresponding prompt information is displayed to prompt entry into the next assembly process.

[0080] Furthermore, the corresponding prompt information includes guidance information of the current assembly process, thereby guiding the user to perform correct assembly operations and ensuring the accuracy of product assembly.

[0081] In one embodiment, the prompt information is generated by the AR device based on the assembly quality inspection result.

[0082] In another embodiment, the prompt information is generated by the server based on the assembly quality inspection result, and the prompt information is sent at the same time as the assembly quality inspection result is sent.

[0083] In another embodiment, when the assembly quality inspection result is a failure, the prompt information is generated by the server based on the assembly quality inspection result; when the assembly quality inspection result is a pass, the prompt information is generated by the AR device based on the assembly quality inspection result. This embodiment mainly takes into account that in the case of failure, the server needs to send guidance information for the current assembly process to guide the user to perform the correct assembly operation, and the corresponding prompt information only needs to prompt to enter the next assembly process, so the prompt information does not need to be generated by the server.

[0084] Furthermore, if the assembly quality test result is a failure, the next assembly process is not allowed; if the assembly quality test result is a pass, the next assembly process is allowed. Based on this, the next assembly process can only be entered when the current assembly process is assembled accurately, thereby improving the accuracy of product assembly and ultimately improving the product qualification rate.

[0085] The product assembly method provided by the embodiment of the present invention displays assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process, so that the assembler can learn the correct operation method of the current assembly process with the help of the AR device. The assembler does not need to be experienced, and only needs to display the assembly operation information to guide the assembler to operate, thereby reducing assembly errors, improving product assembly accuracy, and further improving product qualification rate; after displaying the assembly operation information corresponding to the current assembly process, based on the user operation instruction for the current assembly process, the assembly device operation instruction is sent to the target assembly device, that is, the user can trigger the user operation instruction of the AR device based on the assembly operation information, so that the AR device sends the assembly device operation instruction to the target assembly device, without the user operating the target assembly device by himself, so there is no need to rely on experienced assemblers, thereby improving product assembly accuracy, Ultimately, the product qualification rate is improved; after the assembly operation information corresponding to the current assembly process is displayed, and the user performs the assembly operation of the current assembly process based on the assembly operation information, the assembly quality inspection result sent by the server is received, and the prompt information corresponding to the assembly quality inspection result is displayed, and the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process, and the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process that has been completed, so that in the case of assembly errors, re-assembly can be performed, thereby improving the accuracy of product assembly, and ultimately improving the product qualification rate, and ensuring that inexperienced assemblers can also know whether they can enter the next assembly process, thereby improving product assembly efficiency, and the assembly quality inspection by the server can reduce the computing resources occupied by the AR device, improve the performance of the AR device, and ultimately improve the product assembly quality.

[0086] Based on any of the above embodiments, considering the failure, the server needs to send guidance information of the current assembly process to guide the user to perform the correct assembly operation; based on this, in the method, when the assembly quality inspection result is failure, the above step 130 includes: Receive the assembly quality inspection result sent by the server, and the prompt information corresponding to the assembly quality inspection result.

[0087] Wherein, when the assembly quality inspection result is failure, the prompt information includes guidance information of the current assembly process. The guidance information may include but is not limited to: the location of the unqualified point, the type of the unqualified point and the recommended repair measures for the unqualified point, etc.

[0088] Accordingly, the above step 140 includes: displaying the guidance information by spatially mapping the guidance information into a real environment.

[0089] Here, spatial mapping is to establish anchor points in space by generating a 3D network of the environment, so that the graphical interface can be placed in the physical environment and maintain its position. For example, the guidance information is wiring guidance information. Through spatial mapping technology, the virtual wiring guidance information can be accurately mapped to the real working environment, providing more intuitive and accurate guidance, thereby improving product assembly accuracy and ultimately improving product qualification rate.

[0090] The product assembly method provided by the embodiment of the present invention, when the assembly quality inspection result is a failure of the inspection, receives the assembly quality inspection result sent by the server and the prompt information corresponding to the assembly quality inspection result, so that the prompt information is generated by the server with stronger processing capability, and the prompt information includes the guidance information of the current assembly process, so as to generate more complete guidance information, thereby improving the accuracy of product assembly and ultimately improving the product qualification rate; at the same time, the guidance information is displayed by mapping the guidance information space into the real environment to provide more intuitive and accurate guidance, thereby improving the accuracy of product assembly and ultimately improving the product qualification rate.

[0091] Based on any of the above embodiments, before the above step 130, the method further includes: Collecting a real-time image of the assembly operation of the current assembly process; The real-time image of the assembly operation is sent to the server.

[0092] In one embodiment, only a few real-time images of the assembly operation may be collected.

[0093] In another embodiment, a video stream of the on-site assembly process may be collected first, and the video stream may be sampled to obtain a real-time image of the assembly operation. The sampling may be performed by an AR device or by a server.

[0094] In a specific embodiment, based on the inspection requirements of the current assembly process, real-time images of assembly operations corresponding to the inspection requirements are collected to ensure that whether the assembly is accurate can be accurately determined based on the real-time images of the assembly operations.

[0095] In the product assembly method provided by the embodiment of the present invention, the AR device collects the real-time image of the assembly operation of the current assembly process and sends the real-time image of the assembly operation to the server, thereby eliminating the need for an additional image acquisition device to collect the real-time image of the assembly operation, thereby reducing costs.

[0096] Based on any of the above embodiments, in this method, the assembly operation information is displayed based on the UI interface displayed by the AR device, so as to provide more intuitive and accurate guidance, thereby improving product assembly accuracy and ultimately improving product qualification rate.

[0097] Accordingly, after the above step 120, the method further includes: Based on the user operation instruction, update the UI interface in real time; receiving an execution result corresponding to the assembly device operation instruction returned by the target assembly device; Based on the execution result, the UI interface is updated again.

[0098] Since the user operation instructions are triggered based on the UI interface, the UI interface is updated in real time based on the user operation instructions to prompt the user that the user operation instructions have been responded to, avoiding the user triggering the user operation instructions multiple times, thereby improving the user experience and ensuring the accuracy of product assembly.

[0099] Furthermore, the assembly equipment sends the execution result corresponding to the assembly equipment operation instruction to the server. Furthermore, the server uploads the execution result to the MES system, thereby realizing a closed-loop process.

[0100] The product assembly method provided by the embodiment of the present invention updates the UI interface in real time based on the user operation instruction to prompt the user that the user operation instruction has been responded to, thereby avoiding the user from triggering the user operation instruction multiple times, thereby improving the user experience, and ensuring the accuracy of product assembly, thereby improving the product qualification rate; receives the execution result corresponding to the assembly device operation instruction returned by the target assembly device, and based on the execution result, updates the UI interface again to prompt the user that the user has responded to the user operation instruction, thereby avoiding the user from triggering the user operation instruction multiple times, thereby improving the user experience, and ensuring the accuracy of product assembly, thereby improving the product qualification rate; and prompts the user whether the response is successful based on the execution result, thereby avoiding the user from triggering the user operation instruction multiple times, thereby improving the user experience, and ensuring the accuracy of product assembly, thereby improving the product qualification rate.

[0101] Based on any of the above embodiments, the method further includes: sending the screen image of the AR device to the server, so that the server synchronizes the screen image to a shared terminal.

[0102] Here, the screen image is the image that the user can see through the AR device, based on which the assembly operation screen can be shared. For example, the screen image includes the camera image and UI interface captured by the AR device.

[0103] Among them, the shared terminal includes an AR device and / or a user terminal that supports remote video calls. Based on this, the screen of the current AR device can be shared with another AR device, and can be shared with other user terminals, which is convenient for remote collaboration in product assembly and improves the level of intelligent product assembly. That is, it supports cross-segment web screen projection, so that multiple AR devices can share screen images, and at the same time supports video calls with user terminals (such as mobile phones), which is convenient for remote collaboration and remote guidance.

[0104] For example, in order to better send the screen image of the AR device to the server and synchronize the screen image of the server to the shared terminal, the following processing can be performed. First, use WebRTC (Web Real-Time Communications) technology to integrate the technology into the mixed reality front end to support uploading the screen image of the current AR device, and use the Session Traversal Utilities for NAT (STUN) technology and the Traversal Using Relays around NAT (TURN) technology on the server to build a signaling server, so as to receive the screen projection of the AR device, realize cross-segment web screen projection and video call functions with the mobile phone.

[0105] It should be noted that WebRTC, as a standard and framework for real-time communication technology, integrates a rich set of client APIs (Application Programming Interfaces), audio and video codec libraries, streaming media transmission protocols, echo cancellation, and secure transmission functions, providing a convenient way to achieve low-latency video calls. For example, in the development environment of Hololens, a mixed reality device, the implementation of WebRTC technology relies on the MRTKWebRTC package of Microsoft HoloLens, and uses its stable version to build a real-time communication system. Through the Unity engine, you can import the MRTKWebRTC package and configure related components to achieve audio and video communication functions.

[0106] Considering that WebRTC technology itself does not include a signaling mechanism, a signaling server needs to be built to transmit connection information and data. The embodiment of the present invention uses STUN and TURN to build a signaling server, and installs and starts the server through the command line. The STUN technology is mainly used to help the client discover its public IP address and port; in a NAT environment, devices usually have a private IP address, which makes it difficult for them to establish communication directly with other devices; and the STUN server returns information such as the client's public IP address and port number through interaction with the client; this function is essential for establishing a point-to-point (P2P) connection because it allows devices to exchange necessary network information and try to establish a direct connection. TURN technology is a supplement to STUN technology; TURN technology transmits messages by establishing a relay node, thereby ensuring that reliable real-time communication can be established between devices; when the relay node cannot be established directly, the TURN server will take over the communication process and relay the data packet from one device to another; this mechanism ensures that even if NAT penetration fails, the devices can still maintain communication continuity.

[0107] Specifically, the client will first try to obtain its own public IP address and port through the STUN server, and try to establish a direct connection with the peer. If the connection fails, the client will use the TURN server to establish a relay connection. This flexible mechanism can achieve stable real-time communication in various network environments.

[0108] In the embodiment of the present invention, the MRTK WebRTC package is imported and configured in the AR device, and related components are configured. The coordinated use of these components makes it possible to collect, process and transmit real-time audio and video data. The embodiment of the present invention transmits the camera image and screen UI image collected by the AR device to the signaling server based on STUN and TURN technology through WebRTC technology, and sends it to the client to realize cross-segment web page projection and video call functions with the mobile phone.

[0109] The product assembly method provided by the embodiment of the present invention sends the screen image of the AR device to the server, so that the server synchronizes the screen image to the shared terminal, and the shared terminal includes the AR device and / or a user terminal that supports remote video calls, thereby facilitating remote collaborative product assembly, improving the intelligence level of product assembly, and further improving the accuracy and efficiency of product assembly.

[0110] Based on any of the above embodiments, before the above step 110, the method further includes: Collecting first image information for identifying a user; Sending the first image information to the server; the first image information is used to instruct the server to perform authentication; When the authentication is successful, the assembly operation information sent by the server is received.

[0111] Here, the first image information is information including an image that identifies the user's identity information, for example, the image includes an employee QR code or an employee ID card, etc.

[0112] In a specific embodiment, the first image information is collected by a built-in camera of the AR device.

[0113] It should be understood that the server is allowed to send assembly operation information to the authenticated AR device only when the authentication is successful.

[0114] In one embodiment, when a user uses an AR device for product assembly for the first time (for example, the user starts an application corresponding to the AR device for the first time), a built-in camera of the AR device is called to scan and obtain first image information.

[0115] Furthermore, if the MES (Manufacturing Execution System) information is not used, the server customizes the authentication interface and records the login information of the current user; if the MES information is used, the server's information interface is called.

[0116] For example, Figure 3 As shown, when the user starts the product assembly application of the AR device for the first time, that is, the product assembly application starts running, the built-in camera of the AR device is called to scan the employee QR code, and a login request is made based on the first image information obtained by the scan; the server first determines whether to use the manufacturing execution system information. If not, the server customizes the authentication interface for authentication, and records the current user login record after authentication. If it is used, the server calls the information interface of the MES system. If the call fails, the failure log information is recorded and the return information is encapsulated (the return information is used to return to the AR device). If the call is successful, it detects whether it is the first login. If it is the first login, the current user information is registered to the background system. If it is not the first login, the personnel information is proofread. If the proofreading is successful, the login success information is encapsulated and recorded. If the proofreading fails, the user information is updated; after the login success information is encapsulated and recorded, the electronic work instruction function of the server is started to allow the server to send assembly operation information. It should be noted that the above can be understood as a system login function.

[0117] The product assembly method provided by the embodiment of the present invention collects the first image information used to identify the user's identity, so that the first image information can be collected based on the AR device for user authentication, thereby improving the authentication efficiency, and sending the first image information to the server. The first image information is used to instruct the server to perform authentication. When the authentication is successful, the assembly operation information sent by the server is received to ensure the security of the assembly operation information, that is, it is only provided to the AR device that has successfully authenticated. Therefore, the above method can improve the efficiency and security of product assembly.

[0118] Based on any of the above embodiments, before the above step 110, the method further includes: collecting second image information for identifying the product to be assembled; The second image information is sent to the server; the second image information is used to instruct the server to determine the current product to be assembled.

[0119] Here, the second image information is information including an image identifying information of the product to be assembled, for example, the image includes a product barcode or a product QR code, etc.

[0120] In a specific embodiment, the second image information is collected by a built-in camera of the AR device.

[0121] It should be understood that the second image information is sent to the server so that the server can determine the current product to be assembled, so that the server can determine which product to be assembled whose assembly operation information needs to be returned, and so that the server can better perform assembly quality inspection, thereby improving the accuracy of product assembly and further improving the product qualification rate.

[0122] The product assembly method provided in an embodiment of the present invention collects second image information used to identify the product to be assembled, so that the second image information can be collected based on the AR device to determine the product to be assembled, thereby improving efficiency and reducing costs, and sending the second image information to the server for the server to determine the current product to be assembled, thereby improving the accuracy of product assembly and further improving the product qualification rate.

[0123] Based on any of the above embodiments, the present invention also provides a product assembly method applied to a server. Figure 4 FIG. 3 is a flow chart of a product assembly method according to an embodiment of the present invention. Figure 4 As shown, the product assembly method includes the following steps 410 and 420.

[0124] Step 410 , performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain an assembly quality inspection result.

[0125] Here, the current assembly process is the process step that needs to be assembled currently. The current assembly process may be an assembly process automatically entered from the previous assembly process, or may be an assembly process selected by the assembler through the UI interface of the AR device.

[0126] Here, the real-time image of the assembly operation can be collected in real time by the AR device and sent to the server, or it can be collected in real time by other image acquisition devices (such as a camera) and sent to the server.

[0127] Specifically, the server performs image detection on the real-time image of the assembly operation to determine the assembly quality detection result.

[0128] Furthermore, the assembly quality inspection results are uploaded to the MES system for subsequent review, thus achieving a closed-loop process.

[0129] Step 420: Send the assembly quality inspection result to the AR device.

[0130] The assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process after the assembly operation has been completed.

[0131] Among them, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to the target assembly device based on the user operation instructions for the current assembly process, and the assembly device operation instructions are used to instruct the target assembly device to perform corresponding assembly operations.

[0132] Furthermore, if the assembly quality inspection result is not passed, the assembly quality inspection result is sent to a dedicated display device to display the assembly quality inspection result to assemblers and supervisors in an intuitive manner.

[0133] Furthermore, the server can generate a complete quality report of the product to be assembled based on the results of each assembly quality inspection, thereby ensuring that the assembly quality can be checked later. Furthermore, the quality report can be uploaded to the MES system.

[0134] It should be understood that by sending the assembly quality inspection results to the AR device, if there is an assembly error, it can prompt reassembly, thereby improving the accuracy of product assembly and ultimately improving the product qualification rate.

[0135] In the product assembly method provided by the embodiment of the present invention, the AR device displays the assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process, so that the assembler can learn the correct operation method of the current assembly process with the help of the AR device, and the assembler does not need to be experienced. It only needs to display the assembly operation information to guide the assembler to operate, thereby reducing assembly errors, improving product assembly accuracy, and further improving product qualification rate; after the AR device displays the assembly operation information corresponding to the current assembly process, based on the user operation instruction for the current assembly process, it sends the assembly device operation instruction to the target assembly device, that is, the user can trigger the user operation instruction of the AR device based on the assembly operation information, so that the AR device sends the assembly device operation instruction to the target assembly device, and the user does not need to operate the target assembly device by himself, thereby Without relying on experienced assemblers, the accuracy of product assembly is improved, and the product qualification rate is ultimately improved. The server performs assembly quality inspection on the real-time image of the assembly operation of the current assembly process, obtains the assembly quality inspection result, and sends the assembly quality inspection result to the AR device, so that the AR device displays the prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process that has been completed. In the case of assembly errors, re-assembly can be performed, thereby improving the accuracy of product assembly and ultimately improving the product qualification rate. It also ensures that inexperienced assemblers can also know whether they can enter the next assembly process, thereby improving product assembly efficiency. The assembly quality inspection by the server can reduce the computing resources occupied by the AR device, improve the performance of the AR device, and ultimately improve the product assembly quality.

[0136] Based on any of the above embodiments, Figure 5 FIG. 4 is a flow chart of a product assembly method provided by an embodiment of the present invention. Figure 5 As shown, the above step 410 includes steps 411 to 413.

[0137] Step 411, determining the target object that needs to be detected for the product to be assembled in the current assembly process.

[0138] Here, the product to be assembled is the product that needs to be assembled corresponding to the AR device. The product to be assembled can be determined by scanning a QR code label, or by a method, such as by defaulting to a product on a production line, which is not specifically limited here.

[0139] The target object is the object of the assembly operation required in the current assembly process. The target objects required to be detected in different assembly processes of the product to be assembled may be different. Based on this, the target object required to be detected in the current assembly process of the product to be assembled is first determined, thereby improving the accuracy of subsequent target detection, thereby improving the accuracy of assembly quality detection, and finally improving the accuracy of product assembly.

[0140] For example, it is assumed that the product to be assembled is a compressor, and the target objects to be detected in a certain assembly process of the compressor are plug a and socket A.

[0141] Step 412: Perform target detection on the assembly operation real-time image based on the target object to obtain a target detection result for the target object.

[0142] Here, the target detection result is used to indicate the position and positioning result of the target object. For example, assuming that the product to be assembled is a compressor, the target objects to be detected in a certain assembly process of the compressor are plug model a and socket A, so as to obtain the position of plug a and socket A, which provides a basis for subsequent assembly quality inspection.

[0143] In some embodiments, the real-time image of the assembly operation is input into the target detection model to obtain the target detection result for the target object output by the target detection model. The target detection model is trained based on the sample real-time image of the assembly operation and its corresponding target detection result label. The model parameters and model structure of the target detection model can be continuously iterated and optimized, thereby improving the recognition accuracy and generalization ability of the target detection model for key control points, thereby improving the accuracy and stability of assembly quality detection.

[0144] In one embodiment, a real-time image of the assembly operation of the on-site assembly process is collected as a sample real-time image of the assembly operation. Further, a video stream of the on-site assembly process is first collected, and then the video stream is sampled to obtain the sample real-time image of the assembly operation. Further, the inspection requirements of each assembly process are determined based on the process route of the product to be assembled, so that the corresponding sample real-time image of the assembly operation is collected based on each inspection requirement.

[0145] Furthermore, the collected real-time images of sample assembly operations are preprocessed, and the preprocessing methods may include but are not limited to at least one of the following: denoising, contrast enhancement, color correction, etc., so as to improve the accuracy and stability of assembly quality detection.

[0146] In one embodiment, the target detection model can be constructed based on the YOLOv8 model, thereby improving the accuracy, real-time performance, adaptability to complex scenarios and detection efficiency of target detection, thereby improving the accuracy and efficiency of assembly quality detection.

[0147] Furthermore, the target detection model outputs all detected target information at once based on global image information during the prediction process, without sliding windows of different sizes on the image and then identifying each object through a classifier. During the model training process, the model performs supervised learning through the object information annotated in each image, including the center position, height, width and object category of the object.

[0148] More specifically, image segmentation is first performed, then a bounding box is generated based on a grid, and then post-processing is performed through non-maximal suppression (NMS).

[0149] Furthermore, in the calculation of the loss function of the target detection model, the positive sample allocation strategy of the TaskAligned Assigner was adopted, and the Distribution FocalLoss was introduced, thereby improving the detection accuracy and robustness of the target detection model.

[0150] Furthermore, in terms of data augmentation for sample data, a mosaic enhancement operation was introduced to stitch four images together, forcing the model to learn new positions, partial occlusions, and objects with different surrounding pixels, thereby improving the detection accuracy and robustness of the target detection model.

[0151] Exemplarily, the target detection model is exported as onnx (Open Neural Network Exchange) and integrated in the server, and a background calling script is developed to enable the AR device to communicate with it to realize dynamic detection of real-time images of assembly operations collected in real time.

[0152] Step 413: Perform assembly quality inspection based on the target inspection result to obtain an assembly quality inspection result.

[0153] After obtaining the target detection results, assembly quality inspection can be performed based on rules or based on models.

[0154] For example, assume that the product to be assembled is a compressor, and the target objects to be detected in a certain assembly process of the compressor are plug a and socket A. Based on this, the target detection model identifies the positions of plug a and socket A in the real-time image of the assembly operation; then, it determines whether the wiring is completed. When there is only one socket A in the real-time image of the assembly operation, the criterion for determining whether the wiring is completed is that one plug a is detected in the real-time image of the assembly operation and socket A can no longer be detected. In this case, the system can confirm that plug a has been successfully connected to socket A and the wiring task is completed. When there are multiple sockets A in the image, the multiple sockets A are first sorted in relative position to distinguish them. The criterion for determining whether the wiring is completed is that the number of plugs a in the real-time image of the assembly operation matches that of multiple sockets A and the relative positions are arranged correctly. This standard ensures that in a complex wiring environment, the system can accurately determine the connection status of each plug and the corresponding socket, thereby completing the wiring task.

[0155] The product assembly method provided by the embodiment of the present invention determines the target object that needs to be detected for the product to be assembled in the current assembly process, and the target object is the object of the assembly operation required in the current assembly process, so as to perform target detection on the real-time image of the assembly operation based on the target object, and obtain the target detection result for the target object, so as to more accurately perform assembly quality detection based on the target detection result, and obtain the assembly quality detection result, so as to carry out target detection on the target object that needs to be detected in the current assembly process in a targeted manner, improve the accuracy and efficiency of assembly quality detection, and further improve the accuracy and efficiency of product assembly.

[0156] Based on any of the above embodiments, in this method, the product to be assembled is determined based on the following method: receiving second image information sent by the AR device; the second image information is used to identify the product to be assembled, and the second image information is acquired by the AR device; The product to be assembled is determined based on the second image information.

[0157] Here, the second image information is information including an image identifying information of the product to be assembled, for example, the image includes a product barcode or a product QR code, etc.

[0158] In a specific embodiment, the second image information is collected by a built-in camera of the AR device.

[0159] It should be understood that receiving the second image information sent by the AR device is convenient for the server to determine the current product to be assembled, so that the server can determine which product to be assembled whose assembly operation information needs to be returned, and so that the server can better perform assembly quality inspection, that is, improve the accuracy of assembly quality inspection, thereby improving the accuracy of product assembly, and then improving the product qualification rate.

[0160] The product assembly method provided by an embodiment of the present invention collects second image information for identifying the product to be assembled by an AR device, so that the product to be assembled can be determined based on the second image information collected by the AR device, thereby improving efficiency and reducing costs, and receiving the second image information sent by the AR device for the server to determine the current product to be assembled, thereby improving the accuracy of product assembly and further improving the product qualification rate.

[0161] Based on any of the above embodiments, before the above step 410, the method further includes: Acquire a configured assembly operation information set for a product to be assembled; the assembly operation information set includes information corresponding to the product to be assembled under multiple assembly processes; The assembly operation information corresponding to the current assembly process is sent to the AR device.

[0162] Here, the assembly operation information set includes the assembly operation information corresponding to the product to be assembled under multiple assembly processes, thereby supporting the advance configuration of the assembly operation information set of the product to be assembled, ensuring that the assembly operation information corresponding to the current assembly process can be fed back in time during the assembly process, thereby improving product assembly efficiency.

[0163] It should be understood that the server supports multiple product custom configuration assembly operation information sets, thereby improving the personalization level of product assembly. Further, the assembly operation information set can be uploaded to the MES system, thereby realizing a closed-loop process.

[0164] For example, users can upload the configuration information of product models to the MES system through a specially designed UI interface on the server. The configuration information may include but is not limited to at least one of the following: product specifications, production parameters, and process flow, etc. Based on this, using the multi-level form building function of the web page, a complete operation configuration interface will be automatically generated according to the uploaded configuration information to support multiple product custom configuration assembly operation information sets. The operation configuration interface displays all the configuration details of the product model in an intuitive and easy-to-operate manner, which is convenient for production personnel to quickly understand and execute.

[0165] Furthermore, in order to enhance the richness and intuitiveness of the configuration, the embodiment of the present invention also introduces a file service module. Users can upload multimedia files such as video animations and images required for each assembly process to the MES system. These files will be displayed in an appropriate manner in the operation configuration interface for subsequent upload to the AR device to help production personnel better understand the assembly operation steps and precautions. In addition, this function also provides an MES system access interface, which can realize the configuration of product models, assembly processes, tutorial animations, etc. from the MES system to achieve a closed process loop.

[0166] Furthermore, a server backend system is developed to store and manage data such as 3D reconstruction models, tutorial animations, process requirements, and quality points. At the same time, the server backend system is also responsible for managing data exchange and communication between AR devices and servers to ensure real-time and accurate information transmission. The server backend services of the embodiment of the present invention include product configuration, process configuration, station configuration, assembly equipment operation record query, AR device operation record query, etc., and provide a signaling server carrier for the screen projection and audio and video call functions based on WebRTC technology.

[0167] Furthermore, for the backend server front end, a modern front-end framework is used, using responsive data binding and component-based development to quickly build projects and support single-page application architecture. The entire front end uses JavaScript to achieve full-stack development. In addition, the component library based on Vue.js provides rich UI components and interactive effects, which improves the user experience.

[0168] Furthermore, for the backend of the background server, the springboot framework is used as the overall development framework, which has the advantages of quickly building services, simplifying configuration and improving development efficiency, and integrates multiple modules such as springMVC, springbootWeb, and springData. In terms of functional design, the system provides data support for AR devices through springboot, and uses RabbitMq to achieve decoupling and efficient communication between businesses, which is suitable for processing complex messaging and event-driven scenarios. In database design, the system uses relational database Mysql to store data information, and non-relational database redis to store hot data and user information to achieve minimum authority division. In terms of interface services, the system will be connected to the MES system, verify user identity by scanning work badge information, and interact with assembly equipment. In terms of data processing and communication, the system has designed virtual three-dimensional model loading, AI model calling and WebRTC communication modules to ensure that web pages and AR devices can interact across network segments.

[0169] The product assembly method provided by the embodiment of the present invention obtains the configured assembly operation information set for the product to be assembled, thereby supporting customized configuration of assembly operation information sets for multiple products, thereby improving the personalization level of product assembly and ensuring the accuracy of product assembly; and the assembly operation information set includes information corresponding to the product to be assembled under multiple assembly processes, so as to realize fast sending of the assembly operation information corresponding to the current assembly process to the AR device, that is, supporting the advance configuration of the assembly operation information set of the product to be assembled, ensuring that the assembly operation information corresponding to the current assembly process can be fed back in time during the assembly process, thereby improving the product assembly efficiency.

[0170] Based on any of the above embodiments, in the method, sending the assembly operation information corresponding to the current assembly process to the AR device includes: Receiving first image information sent by the AR device; the first image information is used to identify a user; It is determined that the authentication based on the first image information is successful, and assembly operation information corresponding to the current assembly process is sent to the AR device.

[0171] Here, the first image information is information including an image that identifies the user's identity information, for example, the image includes an employee QR code or an employee ID card, etc.

[0172] In a specific embodiment, the first image information is collected by a built-in camera of the AR device.

[0173] It should be understood that the server is allowed to send assembly operation information to the authenticated AR device only when the authentication is successful.

[0174] In the product assembly method provided by the embodiment of the present invention, the AR device collects the first image information used to identify the user's identity, so that the first image information can be collected based on the AR device for user authentication, thereby improving the authentication efficiency; receiving the first image information sent by the AR device, determining that the authentication based on the first image information is successful, and sending the assembly operation information corresponding to the current assembly process to the AR device, ensuring the security of the assembly operation information, that is, only providing it to the AR device that has successfully authenticated. Therefore, the above method can improve the efficiency and security of product assembly.

[0175] In a specific embodiment, the present invention can realize product assembly based on the mixed reality front end of Microsoft HoloLens, the server back end based on the Ubuntu operating system, and the AI ​​algorithm service. Specifically, the product process route is customized through the server background to realize the real-time ESOP assembly guidance of the AR device, the process AI detection and error prevention, the automatic switching of the process and assembly equipment operation, and the operation record and process closed loop are realized. More specifically, the ESOP is projected onto the AR device in real time using AR augmented reality technology, so that the operator can directly refer to the guidance for assembly and wiring, and the real-time picture captured by the AR device is transmitted to the server background AI interface, and the assembly and wiring quality of the parts are realized online by using AI visual inspection technology to ensure the quality and safety of the product. At the same time, the AR device realizes automatic torque switching by communicating with the torque controller, avoiding human operation errors and quality problems. In addition, every step of the operator's operation will be recorded, realizing the operation record and process closed loop, which is convenient for subsequent quality tracing and problem troubleshooting.

[0176] For example, an electronic work instruction book is displayed through an AR device, which contains detailed steps of the assembly task, and uses a three-dimensional reconstruction model and tutorial animation to intuitively present the position, assembly sequence and tightening torque requirements of the parts. At the same time, the target detection model recognizes and locates the parts in the assembly process in real time to ensure the accurate placement and defect detection of the parts. During the assembly operation, the user sends an operation instruction to the smart tightening gun through the AR device, and the tightening gun performs corresponding actions according to the OpenProtocol protocol, such as starting, stopping and adjusting the tightening torque, and feeds back the execution status to the AR device front end and server in real time. The server backend is not only responsible for storing and managing assembly-related data, but also handles data exchange and communication between the AR device front end and the server to ensure the real-time and accuracy of information. Among them, the Open Protocol protocol is an open communication protocol that is widely used in the control and data exchange of industrial automation equipment. It allows efficient and reliable communication between different devices, and supports equipment status monitoring, operation control and data transmission. Using the Open Protocol protocol, precise control of the tightening gun can be achieved to ensure the efficiency and safety of the assembly process.

[0177] Based on the above embodiments, the present invention realizes efficient and accurate human-machine collaborative assembly work through smooth human-machine interaction and accurate data processing. In addition, the above embodiments have achieved remarkable results in the assembly and wiring of electrical equipment and lines, that is, the wiring efficiency and product qualification rate are improved, and the skill proficiency of wiring personnel is rapidly improved. Specifically, through 3D models and tutorial animations, as well as spatial mapping technology and real-time voice guidance, assemblers can understand and complete wiring tasks more quickly and accurately. At the same time, the intuitiveness and interactivity of AR devices also improve the work enthusiasm and efficiency of assemblers. By performing assembly quality inspection on the real-time images of assembly operations in the current assembly process, quality problems in the wiring process can be discovered and corrected in a timely manner, thereby improving the first-time qualified rate of wiring. The intuitive and visual operation process provided by the AR device also helps assemblers to better perform each step of the operation, reducing the risk of misoperation from the beginning, significantly reducing the rework rate and defective rate, and thus improving the product qualified rate. Through the combined effect of AR-assisted wiring and AI visual quality monitoring and reminder functions, the immersive learning environment and detailed process requirements guidance provided by the AR device enable employees to master wiring skills more quickly. At the same time, the real-time feedback and correction functions of AI visual inspection also accelerate employees' understanding and mastery of quality requirements. In addition, the system can also automatically generate quality reports and data analysis to help employees better understand their skill levels and room for improvement, thereby further promoting skill improvement.

[0178] In summary, the present invention improves the efficiency and accuracy of electrical equipment and line assembly and wiring through intelligent technical means, reduces dependence on skilled workers, and reduces the occurrence of human errors and safety accidents. The invention combines the immersive interactive experience and intelligent recognition capabilities of mixed reality technology to provide a new solution for the wiring of electrical equipment and lines.

[0179] The product assembly device provided by the present invention is described below. The product assembly device described below and the product assembly method described above can be referred to each other.

[0180] Figure 6 is one of the structural schematic diagrams of the product assembly device provided by the embodiment of the present invention, such as Figure 6 As shown, the product assembly device deployed on the AR device includes a first display module 610 , an instruction sending module 620 , a result receiving module 630 and a second display module 640 .

[0181] The first display module 610 is used to display assembly operation information corresponding to the current assembly process; the assembly operation information is used to guide the correct operation method of the current assembly process.

[0182] The instruction sending module 620 is used to send an assembly equipment operation instruction to a target assembly equipment based on a user operation instruction for the current assembly process; the assembly equipment operation instruction is used to instruct the target assembly equipment to perform a corresponding assembly operation.

[0183] The result receiving module 630 is used to receive the assembly quality inspection result sent by the server; the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process.

[0184] The second display module 640 is used to display prompt information corresponding to the assembly quality inspection result; the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process after the assembly operation has been completed.

[0185] Figure 7 FIG. 2 is a second structural diagram of a product assembly device provided by an embodiment of the present invention. Figure 7 As shown, the product assembly device deployed on the server includes a quality detection module 710 and a result sending module 720 .

[0186] The quality inspection module 710 is used to perform assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain an assembly quality inspection result.

[0187] The result sending module 720 is used to send the assembly quality inspection result to the AR device; the assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or prompt to improve the current assembly process after the assembly operation has been completed.

[0188] Among them, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to the target assembly device based on the user operation instructions for the current assembly process, and the assembly device operation instructions are used to instruct the target assembly device to perform corresponding assembly operations.

[0189] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the product assembly method applied to the AR device, the method comprising: displaying the assembly operation information corresponding to the current assembly process; the assembly operation information is used to guide the correct operation mode of the current assembly process; based on the user operation instruction for the current assembly process, sending the assembly equipment operation instruction to the target assembly equipment; the assembly equipment operation instruction is used to instruct the target assembly equipment to perform the corresponding assembly operation; receiving the assembly quality inspection result sent by the server; the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process; displaying the prompt information corresponding to the assembly quality inspection result; the prompt information is used to prompt to enter the next assembly process or prompt to improve the current assembly process that has been completed. Alternatively, a product assembly method applied to a server is executed, the method comprising: performing assembly quality inspection on a real-time image of the assembly operation of a current assembly process to obtain an assembly quality inspection result; sending the assembly quality inspection result to an AR device; the assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, the prompt information being used to prompt entry into the next assembly process or to prompt improvement of the current assembly process that has been completed; wherein, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, the assembly operation information being used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to a target assembly device based on user operation instructions for the current assembly process, the assembly device operation instructions being used to instruct the target assembly device to perform corresponding assembly operations.

[0190] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0191] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the product assembly method applied to AR devices provided by the above-mentioned methods, and the method includes: displaying assembly operation information corresponding to the current assembly process; the assembly operation information is used to guide the correct operation method of the current assembly process; based on the user operation instructions for the current assembly process, sending assembly equipment operation instructions to the target assembly equipment; the assembly equipment operation instructions are used to instruct the target assembly equipment to perform the corresponding assembly operation; receiving the assembly quality inspection result sent by the server; the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process; displaying prompt information corresponding to the assembly quality inspection result; the prompt information is used to prompt to enter the next assembly process or prompt to improve the current assembly process after the assembly operation has been completed. Alternatively, a computer can execute a product assembly method applied to a server provided by each of the above methods, the method comprising: performing assembly quality inspection on a real-time image of the assembly operation of a current assembly process to obtain an assembly quality inspection result; sending the assembly quality inspection result to an AR device; the assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, the prompt information being used to prompt entry into the next assembly process or to prompt improvement of the current assembly process that has been completed; wherein, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, the assembly operation information being used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to a target assembly device based on user operation instructions for the current assembly process, the assembly device operation instructions being used to instruct the target assembly device to perform corresponding assembly operations.

[0192] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the product assembly method applied to an AR device provided by the above-mentioned methods, the method comprising: displaying assembly operation information corresponding to the current assembly process; the assembly operation information is used to guide the correct operation method of the current assembly process; based on the user operation instructions for the current assembly process, sending assembly equipment operation instructions to the target assembly equipment; the assembly equipment operation instructions are used to instruct the target assembly equipment to perform the corresponding assembly operation; receiving the assembly quality inspection result sent by the server; the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process; displaying prompt information corresponding to the assembly quality inspection result; the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process after the assembly operation has been completed. Alternatively, a product assembly method applied to a server provided by the above methods is executed, the method comprising: performing assembly quality inspection on a real-time image of the assembly operation of a current assembly process to obtain an assembly quality inspection result; sending the assembly quality inspection result to an AR device; the assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, the prompt information being used to prompt entry into the next assembly process or prompt improvement of the current assembly process that has been completed; wherein, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, the assembly operation information being used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to a target assembly device based on user operation instructions for the current assembly process, the assembly device operation instructions being used to instruct the target assembly device to perform corresponding assembly operations.

[0193] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the present invention, rather than to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that various combinations, modifications or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and should be included in the scope of the claims of the present invention.

Claims

1. A product assembly method, characterized in that: Applied to an AR device, the method includes: Displaying assembly operation information corresponding to the current assembly process; the assembly operation information is used to guide the correct operation method of the current assembly process; Based on the user operation instruction for the current assembly process, sending an assembly equipment operation instruction to the target assembly equipment; the assembly equipment operation instruction is used to instruct the target assembly equipment to perform a corresponding assembly operation; receiving an assembly quality inspection result sent by a server; the assembly quality inspection result is obtained by the server performing assembly quality inspection on a real-time image of an assembly operation of the current assembly process; The prompt information corresponding to the assembly quality inspection result is displayed; the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process after the assembly operation has been completed.

2. The product assembly method according to claim 1, characterized in that: When the assembly quality inspection result is failure, the receiving assembly quality inspection result sent by the server includes: Receiving the assembly quality inspection result sent by the server, and prompt information corresponding to the assembly quality inspection result; the prompt information includes guidance information of the current assembly process; Accordingly, the prompt information corresponding to the assembly quality inspection result is displayed, including: The guidance information is displayed in a manner of spatially mapping the guidance information onto a real environment.

3. The product assembly method according to claim 1 or 2, characterized in that: Before receiving the assembly quality inspection result sent by the server, the method further includes: Collecting a real-time image of the assembly operation of the current assembly process; The real-time image of the assembly operation is sent to the server.

4. The product assembly method according to claim 1 or 2, characterized in that: The assembly operation information is displayed based on the UI interface displayed by the AR device; After sending the assembly equipment operation instruction to the target assembly equipment based on the user operation instruction for the current assembly process, the method further includes: Based on the user operation instruction, update the UI interface in real time; receiving an execution result corresponding to the assembly device operation instruction returned by the target assembly device; Based on the execution result, the UI interface is updated again.

5. The product assembly method according to claim 1 or 2, characterized in that: The method further comprises: Sending the screen image of the AR device to the server, so that the server synchronizes the screen image to the sharing terminal; The shared terminal includes an AR device and / or a user terminal supporting remote video calls.

6. The product assembly method according to claim 1 or 2, characterized in that: Before displaying the assembly operation information corresponding to the current assembly process, the method further includes: Collecting first image information for identifying a user; Sending the first image information to the server; the first image information is used to instruct the server to perform authentication; When the authentication is successful, the assembly operation information sent by the server is received.

7. The product assembly method according to claim 1 or 2, characterized in that: Before displaying the assembly operation information corresponding to the current assembly process, the method further includes: collecting second image information for identifying the product to be assembled; The second image information is sent to the server; the second image information is used to instruct the server to determine the current product to be assembled.

8. The product assembly method according to claim 1 or 2, characterized in that: The assembly operation information includes a tutorial animation of the current assembly process and assembly operation key information of the current assembly process; the tutorial animation is used to show the correct operation method of the current assembly process, and the assembly operation key information is used to prompt precautions for the assembly operation.

9. A product assembly method, characterized in that: Applied to a server, the method comprises: Performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain the assembly quality inspection result; Sending the assembly quality inspection result to the AR device; the assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or prompt to improve the current assembly process that has been completed; Among them, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to the target assembly device based on the user operation instructions for the current assembly process, and the assembly device operation instructions are used to instruct the target assembly device to perform corresponding assembly operations.

10. The product assembly method according to claim 9, characterized in that: The assembly quality inspection is performed on the real-time image of the assembly operation of the current assembly process to obtain the assembly quality inspection result, including: Determine the target object that needs to be detected for the product to be assembled in the current assembly process; the target object is the object of the assembly operation required in the current assembly process; Performing target detection on the real-time image of the assembly operation based on the target object to obtain a target detection result for the target object; An assembly quality inspection is performed based on the target inspection result to obtain an assembly quality inspection result.

11. The product assembly method according to claim 10, characterized in that: The products to be assembled are determined based on the following method: receiving second image information sent by the AR device; the second image information is used to identify the product to be assembled, and the second image information is acquired by the AR device; The product to be assembled is determined based on the second image information.

12. The product assembly method according to claim 9, characterized in that: Before performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain the assembly quality inspection result, the method further includes: Acquire a configured assembly operation information set for a product to be assembled; the assembly operation information set includes information corresponding to the product to be assembled under multiple assembly processes; The assembly operation information corresponding to the current assembly process is sent to the AR device.

13. The product assembly method according to claim 12, characterized in that: The sending the assembly operation information corresponding to the current assembly process to the AR device includes: Receiving first image information sent by the AR device; the first image information is used to identify a user; It is determined that the authentication based on the first image information is successful, and assembly operation information corresponding to the current assembly process is sent to the AR device.

14. A product assembly device, characterized in that: Deployed on an AR device, the device includes: The first display module is used to display assembly operation information corresponding to the current assembly process; the assembly operation information is used to guide the correct operation method of the current assembly process; An instruction sending module, used for sending an assembly equipment operation instruction to a target assembly equipment based on a user operation instruction for the current assembly process; the assembly equipment operation instruction is used for instructing the target assembly equipment to perform a corresponding assembly operation; A result receiving module is used to receive the assembly quality inspection result sent by the server; the assembly quality inspection result is obtained by the server performing assembly quality inspection on the real-time image of the assembly operation of the current assembly process; The second display module is used to display prompt information corresponding to the assembly quality detection result; the prompt information is used to prompt to enter the next assembly process or to prompt to improve the current assembly process after the assembly operation has been completed.

15. A product assembly device, characterized in that: Deployed on a server, the device includes: The quality inspection module is used to perform assembly quality inspection on the real-time image of the assembly operation of the current assembly process to obtain the assembly quality inspection result; A result sending module is used to send the assembly quality inspection result to the AR device; the assembly quality inspection result is used to instruct the AR device to display prompt information corresponding to the assembly quality inspection result, and the prompt information is used to prompt to enter the next assembly process or prompt to improve the current assembly process that has been completed; Among them, during the assembly process of the current assembly process, the AR device is used to display assembly operation information corresponding to the current assembly process, and the assembly operation information is used to guide the correct operation method of the current assembly process; the AR device is also used to send assembly device operation instructions to the target assembly device based on the user operation instructions for the current assembly process, and the assembly device operation instructions are used to instruct the target assembly device to perform corresponding assembly operations.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the product assembly method according to any one of claims 1 to 13 is implemented.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the product assembly method according to any one of claims 1 to 13 is implemented.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the product assembly method according to any one of claims 1 to 13 is implemented.

Images