Augmented reality cooperation device and system and electronic device

Abstract

The invention provides an augmented reality cooperation device which is suitable for at least one working machine table and comprises a first display assembly, a first photographing assembly and a first operation assembly. The first photographing assembly is used for photographing the at least one working machine to generate a machine image. The first operation assembly is connected to the first display assembly and the first photographing assembly and used for generating a first augmented reality image containing first operation information based on the machine table image and controlling the first display assembly to display the first augmented reality image. And generating a second augmented reality image including second operation information according to the first operation information, another machine image shot at the current time point or a feedback signal of the at least one working machine, and controlling the first display assembly to display the second augmented reality image, the first operation information being different from the second operation information. The augmented reality cooperation device provided by the invention can provide digital teaching materials and guided teaching.

Description

Technical Field

[0001] The present invention relates to augmented reality and remote assistance, and in particular to an augmented reality collaboration device and system. Background Art

[0002] Inspection personnel in factories regularly maintain various equipment for production operations. However, at present, inspection personnel face various inconveniences. First, existing production operation training mostly adopts the master-apprentice training system. However, it is not easy to inherit the experience of the old masters, and oral teaching cannot be effectively absorbed, resulting in the loss of experience. In other words, the inheritance of experience cannot be replicated 100%, and there is an easy problem of experience loss. Moreover, some special procedures require quite professional capabilities, and the operation methods are difficult to replicate. Once the inspection personnel responsible for the special procedures are unable to arrive due to reasons, it is easy to have a shortage of manpower. In addition, when the inspection personnel arrive at the operation site, in the past, personnel needed to check the equipment information through the equipment human-machine interface (HMI), and then log in to the system on the workstation computer to consult the production-related information. The scattered information leads to low operation efficiency. Summary of the Invention

[0003] In view of this, the present invention provides an augmented reality collaboration device and system to solve the above problems.

[0004] An augmented reality collaboration device according to an embodiment of the present invention is applicable to at least one working machine tool, and includes a first display component, a first photographing component, and a first computing component. The first photographing component is used to photograph the at least one working machine tool to generate a machine tool image. The first computing component is connected to the first display component and the first photographing component, and is used to generate a first augmented reality image including first operation information based on the machine tool image, control the first display component to display the first augmented reality image, generate a second augmented reality image including second operation information according to the first operation information, another machine tool image photographed at the current time point, or a feedback signal of the at least one working machine tool, and control the first display component to display the second augmented reality image, wherein the first operation information is different from the second operation information.

[0005] An augmented reality collaboration system according to an embodiment of the present invention includes the augmented reality collaboration device described in the foregoing embodiment and an electronic device. The electronic device includes a second wireless communication module, a second photographing component, and a second display component. The second wireless communication module is used to connect to the first wireless communication module. The second photographing component is used to photograph the working field to generate a field image, wherein the working field is different from the field where the at least one working machine tool is located. The second display component is used to display the machine tool image or the field image.

[0006] In summary, the augmented reality collaboration device and system according to an embodiment of the present invention can provide digital teaching materials and guided teaching. By guiding trainees to perform step-by-step operations through virtual operation information, the learning efficiency is improved and complex operations are decomposed for effective learning.

[0007] The above description of the present disclosure and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and to provide a further explanation of the scope of the patent application of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a block architecture diagram of the augmented reality collaboration device illustrated in the first embodiment of the present invention;

[0009] Figure 2 and Figure 3 It is a schematic example diagram of an augmented reality image including digital billboard information;

[0010] Figure 4 It is a block architecture diagram of the augmented reality collaboration device illustrated in the second embodiment of the present invention;

[0011] Figure 5 It is a block architecture diagram of the positioning module according to an embodiment of the present invention;

[0012] Figure 6 It is a block architecture diagram of the augmented reality collaboration device illustrated in the third embodiment of the present invention; and

[0013] Figure 7 It is a block architecture diagram of the augmented reality collaboration system illustrated in an embodiment of the present invention.

[0014] ELEMENT REFERENCE NUMERAL DESCRIPTION

[0015] 10 First display component

[0016] 30 First photography component

[0017] 50 First computing component

[0018] 70 Positioning module

[0019] 71 Inertial positioning component

[0020] 73 Wireless signal positioning component

[0021] 75 Processing component

[0022] 90 First wireless communication module

[0023] 100, 200, 300 Augmented reality collaboration device

[0024] 400 Electronic device

[0025] 410 Second display component

[0026] 430 Second photography component

[0027] 450 Second wireless communication module

[0028] 1000 Augmented reality collaboration system

[0029] M Working machine

[0030] L1 Digital billboard

[0031] A1, A2 Arrows Detailed implementation manners

[0032] The detailed features and advantages of the present invention are described in detail in the following implementation manners. The content is sufficient for any person skilled in the relevant art to understand the technical content of the present invention and implement it accordingly. And according to the content disclosed in this specification, the scope of the patent application and the drawings, any person skilled in the relevant art can easily understand the relevant objectives and advantages of the present invention. The following embodiments are used to further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention in any way.

[0033] The present invention provides an augmented reality collaboration device and system. The device and system are applicable to at least one working machine in a factory. In one embodiment, the number of the at least one working machine is multiple, but the present invention does not limit the number of working machines. The multiple working machines are arranged in a working field (such as a factory building).

[0034] Please refer to Figure 1 . Figure 1 It is a block architecture diagram of the augmented reality collaboration device shown in the first embodiment of the present invention. As Figure 1 shown, the augmented reality collaboration device 100 includes a first display component 10, a first photography component 30, and a first computing component 50. The first computing component 50 is connected to the first display component 10 and the first photography component 30.

[0035] The augmented reality collaboration device 100 is, for example, a smart phone, a tablet computer, a head-mounted device, or smart glasses (such as Microsoft HoloLens), and the present invention does not limit this.

[0036] The first display component 10 is, for example, the screen of a tablet computer or the lens of a head-mounted device.

[0037] The first photography component 30 is used to photograph the working machine to generate a plurality of machine images. Hereinafter, the machine image taken at the first time point is referred to as the first machine image, and the machine image taken at the second time point is referred to as the second machine image, where the second time point is later than the first time point.

[0038] The first computing component 50 is configured to generate a first augmented reality (AR) image based on the first machine image and control the first display component 10 to display the first augmented reality image, wherein the first augmented reality image includes first operation information. Further, the first augmented reality image may be an integrated image of the first machine image (physical image) and the first operation information (virtual image). The first operation information may provide teaching resources for the user regarding this working machine, including documents, pictures, videos, buttons, icons, forms, steps, etc., but not limited thereto. The first operation information may include digital billboards, which include first operation information such as production status, work order information, equipment information, operation parameters, etc. Please refer to Figure 2 and Figure 3 . Figure 2 and Figure 3 are exemplary schematic diagrams of augmented reality images containing digital billboard information. Figure 2 Shown is the working machine M and the digital billboard L1 showing a certain component of the working machine. Figure 3 Shown is a schematic diagram in which the user can browse the previous step or the next step of the current step through the arrows labeled A1 and A2.

[0039] After the user sees the first augmented reality image presented on the first display component 10, the user operates the current working machine according to the first operation information, and then the working machine generates feedback. The feedback may be in the form of an electronic signal or a machine state that can be photographed. Therefore, the first computing component 50 can generate a second augmented reality image based on the first operation information, the second machine image captured at the second time point, or the feedback signal of the working machine, and control the first display component 10 to display the second augmented reality image, wherein the second augmented reality image includes second operation information. In terms of form, the second operation information is similar to the first operation information. For example, both are teaching videos about the use of the working machine. However, in terms of content, the first operation information is different from the second operation information. For example, the first operation information is the startup procedure of the working machine, and the second operation information is the production procedure of the working machine. The above first operation information and second operation information may be digitized materials copied from the experience of old masters, which can be repeatedly trained and are easy to maintain.

[0040] In an embodiment, the first computing component 50 generates a second augmented reality image when the first operation information is triggered. For example: The user has understood the startup procedure of the working machine through the first augmented reality image and triggers a virtual icon to obtain the description of the production procedure.

[0041] In one embodiment, the first computing component 50 is used to generate a second augmented reality image when it is determined that the second machine image contains a default action completion pattern. For example, after the user turns on the work machine according to the first operation information, the power-on indicator light on the work machine lights up. The first camera component 30 captures the second machine image of the lighted indicator light at the second time point. Therefore, the first computing component 50 can confirm that the user has completed the power-on step based on the above information, and then continue to generate the second augmented reality image to notify the user of the next step.

[0042] In one embodiment, the first computing component 50 generates a second augmented reality image when receiving a feedback signal. The feedback signal is used to indicate that the work machine meets the working status. The feedback signal can be generated by the work machine itself and reported to the augmented reality collaboration device 10 through the Manufacturing Execution System (MES) that manages all work machines. Alternatively, the MES can regularly monitor the working status of the work machine and generate a feedback signal, which is then reported to the augmented reality collaboration device 10.

[0043] In one embodiment, the first computing component 50 matches the captured machine image with a plurality of pre-stored standard machine images based on image recognition technology, thereby confirming which work machine the captured machine image belongs to, and obtaining the first operation information corresponding to the work machine. The plurality of standard machine images can be stored in a cloud server or the augmented reality collaboration device 100 itself, and the present invention is not limited to this.

[0044] In one embodiment, it is necessary to establish a virtual space of the work area where the work machine is located in advance, which includes steps such as icon embedding, virtual environment positioning, message code compilation, event process definition, etc. Then, by scanning the real world work area, the three-dimensional coordinates of the virtual space are integrated with the three-dimensional coordinates of the real world. This includes: real environment positioning, message billboard embedding, icon touch tracking, event guidance, etc.

[0045] Through the above description, it can be known that the augmented reality collaboration device 100 of one embodiment of the present invention can achieve the effect of guided teaching. Augmented reality guided teaching can transform the experience of senior employees into digital teaching materials. The digital environment can guide students to learn step by step through virtual operation information, and the operation information design can set message triggering behavior to complete the operation in conjunction with the system. Students use the method of learning by doing to deepen their learning impression. If they are not proficient in the operation method in the future, they can repeat the learning at any time without relying on others to teach them.

[0046] Please refer to Figure 4 . Figure 4It is a block architecture diagram of the augmented reality collaboration device illustrated in the second embodiment of the present invention. Compared with the first embodiment, the augmented reality device 200 of the second embodiment further includes a positioning module 70. The positioning module 70 is connected to the first computing component 50, where the first display component 10, the first photography component 30, and the first computing component 50 can perform the operations described in the first embodiment and can be implemented by the hardware described in the first embodiment, which will not be elaborated here.

[0047] In the second embodiment, in addition to the operations described in the first embodiment, the first computing component 50 can also obtain the device position information of the augmented reality collaboration device 200 through the positioning module 70. In this embodiment, the number of workstations is multiple. The first computing component 50 is further configured to control the first display component 10 to display the status information of the multiple workstations according to the device position information and the default layout information. The default layout information includes the multiple setting position information of the multiple workstations in the work field. As described above, since the virtual space of the work field where the workstations are located has been established in advance, the work field in the real world is scanned, and the three-dimensional coordinates of the virtual space are integrated with the three-dimensional coordinates of the real world. Therefore, as long as a certain three-dimensional coordinate in the real world is obtained, it is possible to know which workstation the user is currently located at through the above-mentioned virtual-real coordinate correspondence relationship, and then present the status information of the workstation.

[0048] Through the aforementioned system information integration and positioning tracking functions, the position of the inspection personnel can be confirmed in real time. When the personnel arrive at the inspection point, they can view the digital billboard in real time to obtain the key information of the machine.

[0049] Please refer to Figure 5 。 Figure 5 It is a block architecture diagram of the positioning module 70 according to an embodiment of the present invention. As Figure 5 shown, the positioning module 70 includes an inertial positioning component 71, a wireless signal positioning component 73, and a processing component 75. The processing component 75 is connected to the inertial positioning component 71 and the wireless signal positioning component 73.

[0050] The inertial positioning component 71 is configured to generate inertial positioning information. In one embodiment, the implementation of the inertial positioning component 71 can adopt at least one of the following examples: gyroscope, accelerometer, and magnetometer.

[0051] The wireless signal positioning component 73 is used to generate wireless signal positioning information. In one embodiment, the wireless communication standards supported by the wireless signal positioning component 73 may adopt at least one of the following examples: 5th Generation mobile networks (5G), Bluetooth, Wi-Fi, ZigBee, low power wide area network (LPWAN, such as Lora, Sigfox, NB-IoT, etc.).

[0052] The processing component 75 is used to generate device position information according to the inertial positioning information and the wireless signal positioning information. The processing component 75 generates coordinates and direction angles through a hybrid positioning engine.

[0053] Please refer to Figure 6 . Figure 6 It is a block diagram of an augmented reality collaboration device shown in the third embodiment of the present invention. Compared with the first embodiment, the augmented reality collaboration device of the third embodiment further includes a first wireless communication module 90. The first wireless communication module 90 is connected to the first computing component 50, where the first display component 10, the first photography component 30, and the first computing component 50 can perform the operations as described in the first embodiment and can be implemented by the hardware as described in the first embodiment, which will not be elaborated here.

[0054] The first wireless communication module 90 is used to output machine images or receive a remote image. The first computing component 50 is further used to control the first display component 10 to display the remote image. In this embodiment, the effect of augmented reality remote collaboration is achieved through the first wireless communication module 90. Through augmented reality remote collaboration, real-time video and voice communication are provided. Remote personnel can view the remote environment in real time and have a voice call, and through the icon marking tool, operation instructions or discussion items can be clearly indicated, enabling both parties to communicate effectively and avoid operation errors caused by information gaps.

[0055] Please refer to Figure 7 . Figure 7 It is a block diagram of an augmented reality collaboration system shown in an embodiment of the present invention. As Figure 7 shown, the augmented reality collaboration system 1000 includes the augmented reality device 300 of the third embodiment and the electronic device 400. The electronic device 400 includes a second display component 410, a second photography component 430, and a second wireless communication module 450, where the second wireless communication module 450 is connected to the first wireless communication module 90. In one embodiment, the first wireless communication module 90 and the second wireless communication module 450 support a mobile communication network, and this mobile communication network is a 5G private network.

[0056] The second imaging component 430 is configured to capture the working field to generate a field image, where the working field is different from the field where the at least one working machine is located. The second display component 410 is configured to display the machine image or the field image.

[0057] In one embodiment, the first wireless communication module 90 and the second wireless communication module 450 each include a mobile communication technology server key (dongle), such as a 5G dongle. The first wireless communication module 90 outputs a signal with an authentication code to the second wireless communication module 450 through the public network, and when the second wireless communication module 450 confirms that the authentication code is legal, a communication connection is established between the first wireless communication module 90 and the second wireless communication module 450. In one embodiment, the authentication code is the network address (Internet Protocol Address, IP Address) assigned by the 5G private network to the augmented reality device 300 and the hardware identifier of the augmented reality device 300 itself.

[0058] In one embodiment, the implementation of the first computing component 50 and the processing component 75 may adopt at least one of the following examples: a personal computer, a network server, a microcontroller (MCU), an application processor (AP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functions. However, the present invention does not limit the hardware types of the first computing component 50 and the processing component 75.

[0059] The following describes a practical application scenario: the start-up inspection of a Surface-mount technology Printer (SMT Printer). First, a digital billboard can be presented by using a plurality of augmented reality images sequentially generated as the user operates, guiding the user to check the equipment status and production operation requirements. Second, the augmented reality images can be used to connect to the production management system to check whether the stencil and the printing base are qualified tooling. In addition, the augmented reality images can be used to connect to the process management system to confirm the fiducial mark of the stencil sample and the solder landing hole information, and to check whether the component position and the solder landing position meet the production requirements. The user can access digital information, the standard operating procedure or the operation instruction manual of the working machine at any time.

[0060] In summary, the augmented reality collaboration device and system according to an embodiment of the present invention can provide digital teaching materials and guided teaching. By guiding trainees to perform step-by-step operations through virtual operation information, the learning efficiency is improved and complex operations are split for effective learning. In addition, in other embodiments of the present invention, a remote collaboration function can also be provided. Through real-time online connection and video linkage at different locations, it assists in synchronizing the status and collaborating on operations at different locations. In addition, in other embodiments of the present invention, a positioning and inspection function is also provided. By integrating the positioning mechanism, when the user arrives at a fixed point, a digital information billboard is immediately displayed, and production information is integrated for effective navigation so that the user can grasp key information in real time.

[0061] Although the present invention has been disclosed as above in the foregoing embodiments, it is not intended to limit the present invention. Any changes and modifications made without departing from the spirit and scope of the present invention fall within the scope of patent protection of the present invention. For the scope of protection defined by the present invention, please refer to the attached patent application scope.

Claims

1. An augmented reality collaboration device, characterized in that: Applicable to at least one working machine, including: a first display component; a first photographing component, used for photographing the at least one working machine to generate a machine image; and a first computing component connected to the first display component and the first photographing component, for generating a first augmented reality image including a first operation information based on the machine image, controlling the first display component to display the first augmented reality image, generating a second augmented reality image including a second operation information based on the first operation information, another machine image captured at a current time point, or a feedback signal of the at least one working machine, and controlling the first display component to display the second augmented reality image; The first operation information is different from the second operation information.

2. The augmented reality collaboration device according to claim 1, characterized in that: The first computing component generates the second augmented reality image when the first operation information is triggered.

3. The augmented reality collaboration device according to claim 1, characterized in that: The first computing component is used to generate the second augmented reality image when it is determined that the another machine image includes a default action completion pattern.

4. The augmented reality collaboration device according to claim 1, characterized in that: The feedback signal indicates that the at least one working machine meets a working state, and the first computing component generates the second augmented reality image when receiving the feedback signal.

5. The augmented reality collaboration device according to claim 1, characterized in that: The at least one working machine is multiple in number and is disposed in a working area, and the augmented reality collaboration device further comprises: a positioning module, connected to the first computing component, for generating device location information; The first computing component is further used to control the first display component to display a status information of one of the at least one work machine according to the device position information and a default arrangement information, and the default arrangement information includes a plurality of setting position information of the at least one work machine in the work field.

6. The augmented reality collaboration device according to claim 5, characterized in that: The positioning module comprises: An inertial positioning component, used to generate inertial positioning information; A wireless signal positioning component, used to generate wireless signal positioning information; and A processing component is connected to the inertial positioning component and the wireless signal positioning component, and is used to generate the device position information according to the inertial positioning information and the wireless signal positioning information.

7. The augmented reality collaboration device according to claim 1, characterized in that: Also includes: a first wireless communication module, connected to the first computing component, for outputting the machine image or receiving a remote image; The first computing component is further used to control the first display component to display the remote image.

8. The augmented reality collaboration device according to claim 7, characterized in that: The first wireless communication module includes a mobile communication technology server key.

9. An augmented reality collaboration system, characterized in that: Include: The augmented reality collaboration device as claimed in claim 7; and An electronic device comprising: a second wireless communication module, used to connect to the first wireless communication module; a second camera assembly, used for photographing a working field to generate a field image, wherein the working field is different from the field where the at least one working machine is located; and A second display component is used to display the machine image or the field image.

10. The augmented reality collaboration system according to claim 9, characterized in that: The first wireless communication module and the second wireless communication module each include a mobile communication technology server key. The first wireless communication module outputs a signal with an authentication code to the second wireless communication module through a public network, and when the second wireless communication module confirms that the authentication code is legal, a communication connection is established between the first wireless communication module and the second wireless communication module.

Images