A virtual inspection system and a visual factory system in an augmented reality environment

Abstract

A virtual inspection system and a visualized factory system in an augmented reality environment, comprising at least a server system (1), at least one interactive terminal and at least one control system (60), wherein the server system (1) establishes communication connections with the at least one interactive terminal and the at least one control system (60), respectively, and the server system (1) comprises at least a two-dimensional module (10) for displaying a two-dimensional graphic component of a controlled object (70) and a three-dimensional module (20) for displaying a three-dimensional graphic component of the controlled object (70), wherein a mapping relationship is established between the two-dimensional graphic component and the three-dimensional graphic component of the controlled object (70), and in response to an inspection personnel operating the two-dimensional view at the interactive terminal, the three-dimensional module (20) can locate the three-dimensional graphic component corresponding to the specified two-dimensional graphic component in the three-dimensional view.

Description

Technical Field

[0001] The present invention relates to the field of virtual control technology, and in particular to a virtual inspection system and a visualized factory system in an augmented reality environment. Background Art

[0002] Digital twins have evolved from technologies such as virtual manufacturing and digital prototypes. They were originally proposed by the U.S. military in aerospace system research and have now expanded to multiple fields such as intelligent manufacturing, predicting equipment failures, and improving products. Digital twins are also used to refer to the completion of digital models of a factory's workshop and production line before they are built. In this way, the factory is simulated and simulated in a virtual cyberspace, and the real parameters are passed to the actual factory construction. After the workshop and production line are built, the two continue to exchange information in daily operation and maintenance. However, the current virtual system of the digital twin factory only virtualizes the equipment operation of the factory, and does not specifically include the inspection status of the inspectors in the digital twin virtual system, so that the inspectors can only view the surface physical characteristics of the system during the inspection, and cannot view the real-time data of the equipment at any time during the inspection.

[0003] For example, Chinese patent CN108628595A discloses a system and method for developing a control application for a controller used in an automation system. During the execution of the control application, the controller receives a sensor signal associated with the perception of a first real component. The activity of the virtual component is simulated, and the activity includes interaction with the first real component. The virtual component is a digital twin of a second real component designed for a working environment and does not exist in the working environment. Virtual data is generated in response to the simulated activity of the virtual component. The control application module uses the sensor signal and virtual data for the first real component to determine the parameters for developing the control application. During the execution of the control application, an augmented reality display signal for the working environment is rendered and displayed based on the digital representation of the virtual data. However, the defect that still exists in this invention is that when the inspection personnel walk in the factory area with wearable devices, the ordinary data request algorithm is unstable and cannot accurately provide information, which makes the system unstable.

[0004] The prior art also discloses a method of implementing inspections through augmented reality devices. Augmented reality is the intersection where virtual and reality meet. AR can superimpose virtual images on real objects and provide all necessary data for staff performing tasks. Staff can know which specific part of the equipment has failed without guessing, and can use augmented reality technology to present the part within the field of vision to determine whether it has failed. In addition to making product production more efficient, wearable devices can also help workers avoid crises and protect their health and safety. For example, Chinese patent CN111176245A discloses a multi-terminal industrial equipment inspection and monitoring system and method based on digital twin technology, including a physical entity module, a visual interaction module, a data calculation module, and a data transmission and feedback drive module. The invention extracts all the status data of the industrial equipment production process through PLC, and then transmits the key data to the server through the OPC UA interface, and uses cloud services to process the data, and cooperates with augmented reality devices, PCs and mobile devices to achieve the mapping of physical space to digital space, and the feedback control of digital space to physical space, so as to achieve the visual inspection of the production status of industrial equipment, solve the problems of poor data visualization and poor correlation between data in the production process, and significantly improve the data visualization and real-time data feedback in the industrial manufacturing process, so as to achieve the purpose of early warning of faults and real-time visualization of equipment data. The invention obviously inspects the equipment through the system, rather than viewing the equipment data within the set range in a roaming manner in a three-dimensional scene.

[0005] In addition, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making the present invention, but due to space limitations, not all details and contents are listed in detail. However, this does not mean that the present invention does not have the characteristics of these prior arts. On the contrary, the present invention already has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art to the background technology. Summary of the invention

[0006] The defects of the devices or systems that use augmented reality to control and visualize controlled objects in the prior art are that, first, in the existing devices that control visualized objects through augmented reality devices, the inspectors wearing augmented reality devices can only determine their own positions based on positioning information, but cannot confirm their relative positions in the factory; second, the inspectors wearing augmented reality devices cannot view the two-dimensional view and three-dimensional view of the controlled object at the same time, nor can they display the depth of field of the three-dimensional view through the area setting of the two-dimensional view. In the prior art, when inspectors inspect controlled objects in the factory, they can only read the relevant information of the controlled objects and their three-dimensional graphic components around them, and inspect the controlled objects in turn according to a certain route. Most of the controlled objects will not have problems during the inspection. This inspection method will reduce the efficiency of the inspection and cannot detect problematic controlled objects in advance. How to improve the inspection efficiency of inspectors is an unresolved technical problem.

[0007] In view of the deficiencies of the prior art, the present invention provides a virtual inspection system, which at least includes a server system, at least one interactive terminal and at least one control system. The server system establishes communication connections with at least one interactive terminal and at least one control system respectively. The server system at least includes a two-dimensional module for displaying a two-dimensional graphic component of a controlled object and a three-dimensional module for displaying a three-dimensional graphic component of a controlled object. A mapping relationship is established between the two-dimensional graphic component of the controlled object and the three-dimensional graphic component. In response to the operation of the two-dimensional view by the inspection personnel on the interactive terminal, the three-dimensional module can locate the three-dimensional graphic component corresponding to the specified two-dimensional graphic component in the three-dimensional view. Through the corresponding display of the two-dimensional view and the three-dimensional view of the interactive terminal, the present invention only needs to select the controlled object in one of the views to simultaneously view the two-dimensional and three-dimensional views of the controlled object.

[0008] Preferably, the properties of the controlled object can be displayed in parallel in two-dimensional and three-dimensional modes on the interactive terminal in response to the operation of the inspection personnel, and the two-dimensional module and the three-dimensional module simultaneously change the two-dimensional graphic components and three-dimensional graphic components of the controlled object based on the property changes of the controlled object. Compared with the augmented reality device in the prior art that can only display a two-dimensional view or a three-dimensional view, the parallel working mode of the two-dimensional module and the three-dimensional module of the present invention does not occupy the control network bandwidth, has a small workload, reduces the data delay display phenomenon of the three-dimensional model server and the two-dimensional module, and significantly improves the working efficiency of the graphic display, thereby realizing the synchronous real-time display of two-dimensional graphics and three-dimensional graphics.

[0009] Preferably, in response to the depth of field area specified by the inspector in the two-dimensional view, the three-dimensional module sends the data of the three-dimensional graphics of the equipment located in the depth of field area to the interactive terminal, so that the three-dimensional view displays the three-dimensional graphics of the equipment located in the depth of field area in a clear manner, and the three-dimensional view displays the three-dimensional graphics of the equipment located outside the depth of field area in an unclear manner. Such a setting enables the inspectors to conduct inspections in a focused manner during the inspection process, without turning back due to missing the inspection target, and without generating psychological pressure due to unclear relative distances between themselves and the inspection target. The inspectors can clearly understand their own regional location in the factory through the two-dimensional view, and determine the information of the controlled objects around their own location through the three-dimensional view.

[0010] Preferably, when the inspection range viewed by the inspection personnel approaches or goes out of the edge of the depth of field area, the interactive terminal sends a reminder message to remind the personnel that the inspection range has deviated from the inspection range of the specified depth of field area.

[0011] Preferably, the server system further comprises a graphics server, which is used to store the attribute data, digital engineering data and graphic engineering data of the controlled object sent by the control system, and the graphics server fuses the real-time attribute data of the controlled object with the graphic data to form a three-dimensional graphic component capable of three-dimensional display, and displays the attributes of the controlled object in a three-dimensional view. Such a setting is conducive to reducing the workload and processing amount of the three-dimensional module, and is conducive to the three-dimensional module displaying the three-dimensional graphic component faster.

[0012] Preferably, in response to the interactive operation of the inspection personnel on the interactive terminal, the two-dimensional view of the controlled object jumps to the corresponding three-dimensional view.

[0013] Preferably, the inspection personnel can delineate the depth of field area containing the alarm device and its associated devices on the two-dimensional view of the interactive terminal, and view the three-dimensional graphics and attribute information of the alarm device and its associated devices in detail in high definition on the three-dimensional view of the interactive terminal.

[0014] The present invention also provides a virtual inspection method, which at least includes: establishing a mapping relationship between the two-dimensional graphic components and the three-dimensional graphic components of the controlled object, and in response to the inspection personnel's operation on the two-dimensional view on the interactive terminal, the three-dimensional module can locate the three-dimensional graphic component corresponding to the specified two-dimensional graphic component in the three-dimensional view.

[0015] Preferably, the method also includes: in response to the inspection personnel's operation on the controlled object on the interactive terminal, the properties of the controlled object can be displayed in parallel on the interactive terminal in two-dimensional and three-dimensional modes, and the two-dimensional module and the three-dimensional module simultaneously change the two-dimensional graphic components and the three-dimensional graphic components of the controlled object based on the changes in the properties of the controlled object.

[0016] Preferably, the method also includes: in response to the inspection personnel specifying the depth of field area in the two-dimensional view, the interactive terminal receives and displays the three-dimensional graphics of the equipment located in the depth of field area sent by the three-dimensional module, wherein the three-dimensional view displays the three-dimensional graphics of the equipment located in the depth of field area in a clear manner, and the three-dimensional view displays the three-dimensional graphics of the equipment located outside the depth of field area in an unclear manner.

[0017] Preferably, the method further comprises: when the inspection range viewed by the inspection personnel approaches or goes out of the edge of the depth of field area, the interactive terminal sends a reminder message to remind the personnel that the inspection range deviates from the inspection range of the specified depth of field area.

[0018] In the inspection method of the present invention, by specifying the depth of field area, the three-dimensional module can reduce the amount of data transmitted to the interactive terminal for viewing. The three-dimensional module selects the high-definition data of the three-dimensional graphics and their attribute data based on the depth of field area and the device list in the depth of field area sent by the two-dimensional module, and sends them to the interactive terminal. That is, the three-dimensional module can filter and transmit data in a targeted manner based on the depth of field area, thereby reducing the transmission of a large amount of unnecessary data.

[0019] The present invention also provides a visualized factory system in an augmented reality environment, which comprises at least an augmented reality device, a server system and a number of controlled objects, wherein the server system comprises at least a two-dimensional module and a three-dimensional module, wherein the two-dimensional module displays the two-dimensional graphic components of the controlled object in a two-dimensional manner; the three-dimensional module displays the three-dimensional graphic components of the controlled object in a three-dimensional manner; the attribute data of the controlled object can be displayed in parallel in the form of a two-dimensional graphic and a three-dimensional view in response to the operating instructions of the augmented reality device, and the two-dimensional module and / or the three-dimensional module can control the controlled object based on the operating instructions fed back by the augmented reality device.

[0020] Preferably, in response to a definition operation of a depth of field region of a two-dimensional view of the controlled object, the three-dimensional module sends and displays a three-dimensional graphic component of the controlled object within the depth of field region to the augmented reality device.

[0021] Preferably, the three-dimensional graphic component is formed by a graphic server fusing graphic data and attribute data, and is sent to the three-dimensional module in the form of a picture stream, and the three-dimensional module displays the received three-dimensional graphic component in a rendering manner.

[0022] Preferably, in response to the operation information of the action characteristics of the augmented reality device, the three-dimensional module determines the controlled object that needs to be viewed and / or controlled based on the forward direction and action characteristics of the inspection personnel.

[0023] The visualized factory system in the augmented reality environment of the present invention takes advantage of the parallel display of two-dimensional modules and three-dimensional modules, so that the inspection personnel can combine online inspection and offline inspection, which saves physical strength and can also inspect the controlled objects in a real way. When the inspection personnel need to retrieve relevant information about the controlled objects around them, the inspection personnel can select the controlled objects and obtain relevant information based on the relative position of the equipment in the two-dimensional view and the features of the three-dimensional graphic components corresponding to the controlled objects in the three-dimensional view, without the need for the augmented reality device to determine the controlled objects by collecting information about the controlled objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a structural schematic diagram of a virtual inspection system of the present invention;

[0025] Figure 2 is another structural schematic diagram of the virtual inspection system of the present invention;

[0026] Figure 3 It is one of the inspection diagrams roaming in a three-dimensional scene;

[0027] Figure 4 It is another inspection diagram for roaming in a three-dimensional scene;

[0028] Figure 5 It is a schematic diagram of the parallel display of the two-dimensional view and the three-dimensional view of a controlled object;

[0029] Figure 6 It is a logical structure diagram of the present invention;

[0030] Figure 7 It is another logical structure schematic diagram of the present invention;

[0031] Figure 8 It is a two-dimensional schematic diagram of one of the controlled objects of the present invention.

[0032] Reference numerals list

[0033] 1: server system; 10: two-dimensional module; 20: three-dimensional module; 30: graphic server; 40: first interactive terminal; 50: second interactive terminal; 60: control system; 70: controlled object; 80: augmented reality device; 81: two-dimensional operator station; 82: three-dimensional operator station. DETAILED DESCRIPTION

[0034] The following is a detailed description with reference to the accompanying drawings.

[0035] The invention discloses a virtual inspection system and a visualized factory system in an augmented reality environment.

[0036] The picture frequency stream in the present invention refers to: a video stream that can form a picture. Video stream refers to the transmission of video data, for example, it can be processed as a stable and continuous stream through a network.

[0037] The virtual inspection system of the present invention enables inspection personnel to conduct virtual inspections of digital factories through a real perspective of roaming in a three-dimensional scene. The virtual three-dimensional scene of the digital factory of the present invention is consistent with the architectural structure, equipment layout, and equipment connection relationship of the real scene of the physical factory. That is, the three-dimensional graphic components displayed by the three-dimensional scene of the present invention can display the properties of the controlled object. When the attribute data of the controlled object changes, the attributes of the three-dimensional graphics of the controlled object change synchronously, so that the inspection personnel can conduct virtual inspections in a roaming manner in a three-dimensional scene consistent with the real scene, and view the attribute data of the controlled object within a set range.

[0038] The two-dimensional module 10 and the three-dimensional module 20 can be one or more of a dedicated integrated chip, a CPU processor, a server, a server group, and a cloud server. The communication connection in the present invention can be a wired communication connection or a wireless communication connection. The control system 60 can be one or more of a dedicated integrated chip, a processor, a server, and a control module with a control function.

[0039] Example 1

[0040] like Figure 1 As shown, the virtual inspection system of the present invention at least includes an interactive terminal, a server system and a control system 60. The interactive terminal establishes a communication connection with the server system. The server system establishes a communication connection with the control system 60. The control system 60 establishes a communication connection with a number of controlled objects 70, thereby monitoring the attribute data and operation data of the controlled objects and controlling the controlled objects. The control system 60 includes at least a control processor and a control device, which can control the controlled objects in response to the instructions of the interactive terminal. The controlled objects of the present invention include the building structure, engineering equipment, process system, electrical system, instrumentation and control system and communication system of the factory.

[0041] The interactive terminal includes at least one interactive device. The interactive device is, for example, an operator station, an interactive mobile terminal, and an intelligent wearable device. The interactive mobile terminal is, for example, a smart phone, a mobile computer, an iPad, and the like. When the interactive terminal display screen is displayed alone, the interactive terminal can display a two-dimensional graphic and / or a three-dimensional graphic of the controlled object. When the display screen of the interactive terminal can be displayed in split screens, or when there is more than one display screen, for example, the interactive terminal includes a first interactive terminal 40 and a second interactive terminal 50, the first interactive terminal 40 displays a two-dimensional graphic of the controlled object, and the second interactive terminal 50 displays a three-dimensional graphic. The two-dimensional graphic and the three-dimensional graphic are displayed in parallel.

[0042] Each component of the controlled object 70 is encoded according to a predefined unified encoding rule. Preferably, the encoding of each component is a unique interface for calling each component. By using the encoding as the calling object when needed, the present invention does not need to perform relationship retrieval, analysis, screening and other operations on the data, and the three-dimensional graphics of the controlled object can be retrieved quickly and efficiently.

[0043] like Figure 1-2 As shown, the server system of the present invention includes at least a two-dimensional module 10 and a three-dimensional module 20. The two-dimensional module 10 is used to display the controlled object in a two-dimensional manner so that the inspection personnel can control its status. At the interactive terminal, the operator can operate the two-dimensional graphic component that displays any controlled object. Among them, the number, function, purpose, set value, input / output data, alarm and related various engineering design graphics, data and other information of the controlled object are defined in the two-dimensional module. The two-dimensional module can inform the operator of engineering accidents in the form of sound and light alarms by displaying the parameters and properties of all variables in the control process, and can display real-time or historical trend graphs of engineering data in any combination or single-point mode, and can know the engineering safety interlock protection status, etc. Preferably, the same controlled object in the two-dimensional module and the three-dimensional module uses the same code, which is conducive to the synchronous display and control of the same controlled object by the two-dimensional module and the three-dimensional module.

[0044] The three-dimensional module 20 is used to display the controlled object in a three-dimensional manner so that the inspection personnel can display the controlled object they are currently controlling or viewing, such as Figure 3-4 Specifically, at the interactive terminal, the operator can operate the three-dimensional graphic component to display any controlled object. Preferably, the three-dimensional graphic component of the present invention is used to display the properties of the controlled object. That is, the three-dimensional graphic component of the present invention can change synchronously based on the property changes of the controlled object, thereby displaying the properties of the controlled object in real time.

[0045] Preferably, the server system also includes a graphics server 30. The graphics server is used to store the attribute data, digital engineering data and graphic engineering data of the controlled object sent by the control system. The digital engineering data in the prior art only contains pure digital data information and lacks graphic information, which results in the need to adjust the graphics according to the engineering data when displaying two-dimensional or three-dimensional graphics, thereby causing two-dimensional or three-dimensional graphics to be displayed or converted delayed or even stuck. The graphic engineering data of the present invention includes graphic data, real-time attribute data and real-time personnel positioning data. The graphic data of the present invention includes a variety of graphic data such as factory structure graphics, controlled object graphics, and connected device graphics. Specifically, the graphic data also includes flow charts, equipment manufacturing drawings, factory building graphics, controlled object two-dimensional contour graphics, controlled object three-dimensional contour graphics, personnel sign graphics, etc. The factory building graphics include two-dimensional plane graphics and three-dimensional graphics that are consistent with the proportion of the factory physical building structure. Real-time personnel positioning data is the real-time position data of personnel carrying a positioning device. Real-time engineering data, such as various instruments and equipment, are used to collect and monitor changes in process parameter values ​​such as temperature, pressure, flow, and liquid level; the open or closed state of valves; the start or stop state of motors; real-time data of the physical properties of a certain medium; the concentration of combustible gas monitored by combustible gas detection probes; the concentration of toxic gas monitored by toxic gas detection probes; real-time video images monitored by on-site cameras, etc. It also includes various types of actuators, such as control valves, switch valves, actuators, electrical control circuits, etc.

[0046] The graphic server fuses the real-time attribute data of the controlled object with the graphic data to form a three-dimensional graphic component capable of three-dimensional display, and displays the attributes of the controlled object in a three-dimensional view. Alternatively, the graphic server fuses the real-time attribute data of the controlled object with the graphic data to form a two-dimensional graphic component capable of two-dimensional display, and displays the attributes of the controlled object in a two-dimensional view. Figure 3 to Figure 5 As shown, the inspection personnel can view the controlled object from various viewing angles through the interactive terminal. Preferably, the three-dimensional graphics are lightweight processed on the graphics server or the third-party server, and the lightweight processed three-dimensional graphics data are merged with the real-time attribute data. Such a setting is conducive to reducing the workload and processing volume of the three-dimensional module, and is conducive to the three-dimensional module displaying the three-dimensional graphic components faster. Preferably, the three-dimensional module of the present invention displays the three-dimensional graphic components in the form of a picture frequency stream from the graphics server.

[0047] The three-dimensional module responds to the request of the second interactive terminal 50 displaying the three-dimensional graphic component, and displays the properties of the controlled object at the corresponding angle in a roaming manner. Preferably, the three-dimensional module displays the three-dimensional graphic component of the controlled object at a preset initial angle, and adjusts the display angle of the controlled object based on the operation of the inspection personnel.

[0048] The attribute data of the present invention includes self-attribute data, working state attributes, control attributes, environmental attributes, early warning attributes, etc. The self-attribute data is the data of the controlled object itself, such as model, size, location, replaceable reserve quantity, installation time, maintenance time and other information. The working state attribute is the working state data information of the controlled object, such as the temperature, pressure value, liquid level, outlet flow, and real-time data of the physical property components of the outlet medium in a certain controlled object. The control attribute is the control state, control operation execution time, control effect and other information, such as the opening or closing of the valve, the start or stop of the motor, etc. The environmental attribute is the data information related to the environment, such as the concentration monitored by the combustible gas detection probe; the toxic gas concentration monitored by the toxic gas detection probe; the real-time video screen monitored by the on-site camera, etc. The early warning attribute is the attribute data of the controlled object reaching the threshold or operating state that requires an alarm. The operating state is, for example, abnormal working state, flow channel blockage, valve jam, valve failure, abnormal temperature, abnormal pressure, etc.

[0049] Preferably, the two-dimensional graphic component of the controlled object is associated with the three-dimensional graphic component of the controlled object in a mapping manner, so that the inspection personnel can locate the two-dimensional graphic component in the three-dimensional view. Figure 8 As shown, specifically, when the inspector clicks on a certain component on the display of the first interactive terminal 40, the two-dimensional module responds to the inspection personnel's pre-setting of the two-dimensional graphic component by clicking on the two-dimensional graphic component of the controlled object, and sends the display request information of the controlled object to the three-dimensional module. In response to the display request information, the three-dimensional module displays the three-dimensional graphic component corresponding to the two-dimensional graphic component of the controlled object according to the pre-setting. For example, when the inspector clicks on the two-dimensional graphic component of a controlled object at the first interactive terminal 40, the display device of the second interactive terminal 50 displays the three-dimensional graphic component of the controlled object. Preferably, the three-dimensional module displays the three-dimensional graphic component of the controlled object on the second interactive terminal 50 in a rendering manner. Among them, the color of the clicked controlled object is displayed in a highlighted manner or in a color changing manner, so that the inspector can quickly determine the controlled object. The present invention enables the inspector to conveniently locate the two-dimensional graphics and the three-dimensional graphics with each other through the setting of mapping association, so that the inspector can view the two-dimensional view and the three-dimensional view of the controlled object at the same time.

[0050] Preferably, the first interactive terminal 40 and the second interactive terminal 50 display the controlled object in two-dimensional and three-dimensional parallel modes. When the properties of the controlled object change, the two-dimensional module and the three-dimensional module simultaneously change the properties of the two-dimensional graphic components and the three-dimensional graphic components of the controlled object. Compared with the serial working mode of the two-dimensional module and the three-dimensional module in the prior art, the parallel working mode of the two-dimensional module and the three-dimensional module of the present invention does not occupy the control network bandwidth, has a small workload, reduces the data delay display phenomenon of the three-dimensional module and the two-dimensional module, and significantly improves the working efficiency of the graphic display, thereby realizing the synchronous real-time display of the two-dimensional graphics and the three-dimensional graphics.

[0051] Preferably, in response to the interactive operation of the inspection personnel on the interactive terminal, the two-dimensional view of the controlled object can jump to the corresponding three-dimensional view. Preferably, the switching method of the two-dimensional graphic component and the three-dimensional graphic component in the present invention at least includes: point switching, coding switching and switching based on alarm information. The switching method of the two-dimensional graphic component and the three-dimensional graphic component of the present invention is not limited to the listed methods, and can also include other methods that can realize the switching of two-dimensional and three-dimensional graphics, for example, switching in response to the coded content in the form of voice, switching in response to the coded information in the video or picture of the smart wearable device, etc. The point switching method is, for example, by clicking on the device or component on the two-dimensional screen, locating the associated device displayed on the three-dimensional screen. The coding switching method is, for example, directly inputting the code of the controlled object at the corresponding position on the two-dimensional screen to locate the associated device of the controlled object on the three-dimensional display screen. The switching method based on alarm information is, for example, after the control system issues an alarm information related to the controlled object, an alarm information list is formed. The three-dimensional graphic component of the controlled object where the alarm occurs is located by clicking on the alarm list.

[0052] Preferably, when the first interactive terminal 40 displays a two-dimensional view, the three-dimensional module can jump and display the three-dimensional graphics of the controlled object on the first interactive terminal in response to the switching operation of the inspector, and can also jump and display the three-dimensional graphics of the controlled object on the second interactive terminal 50. The present invention can realize the jump from the three-dimensional graphic component of the controlled object to the two-dimensional graphic component of the same controlled object based on the same principle. For example, when the inspector inspects a certain controlled object from the perspective of roaming inspection through the second interactive terminal 50, the properties of the corresponding component can be viewed and the corresponding component can be controlled by clicking on the three-dimensional graphic component of the controlled object. When the inspector clicks the operation of switching the two-dimensional graphics, the two-dimensional module responds to the inspection request of the inspector and displays the two-dimensional graphic component of the controlled object on the first interactive terminal 40. Preferably, the first interactive terminal 40 can display the virtual position of the inspector in the two-dimensional view of the controlled object in response to the roaming perspective of the inspector, so that the inspector can more conveniently view the roaming location. Compared with the prior art, which requires workers to conduct inspections in physical factories, the operators of the present invention can conduct virtual inspections of controlled objects in a three-dimensional view in an interactive terminal, and can view the controlled objects and real-time properties at any time during the inspection and perform real-time control. For controlled objects that issue alarm information, the inspection personnel of the present invention can quickly identify abnormal controlled objects and view the abnormal properties, and even perform corresponding control on abnormal controlled objects through the interactive terminal. Therefore, the virtual inspection system of the present invention can save inspection time, achieve the same working effect as real inspection, and is also conducive to improving inspection efficiency.

[0053] In the prior art, the number of controlled objects in a factory is huge. Since there is no reference to a two-dimensional view, the inspectors search and view the controlled objects based on the connection relationship, which can easily blur the main target of the inspection, resulting in an increasing number of controlled objects being inspected, which reduces the efficiency of the inspection. The three-dimensional view of the present invention can display three-dimensional graphic components within a preset range containing the specified controlled objects, which is conducive to inspectors viewing the controlled objects through an interactive terminal and inspecting the controlled objects. Among them, the three-dimensional view of the present invention requests data according to the depth of field range specified in the two-dimensional scene, and gives a three-dimensional view of the controlled objects and the equipment associated with the controlled objects within the depth of field range. The depth of field range of the present invention refers to the display range that can form a clear image on the display when obtaining a view of a limited distance. The depth of field area refers to the distance range that can clearly display the three-dimensional graphic components of the controlled objects.

[0054] The area delineated on the two-dimensional view may be a specified depth of field range. The depth of field range includes a number of controlled objects. When the inspector delineates the depth of field area, the three-dimensional view displays the three-dimensional graphic components of the controlled object according to the specified depth of field range. For example, the inspector delineates the depth of field area of ​​the three-dimensional view with an undefined geometric shape on the two-dimensional view displayed on the interactive terminal, and the two-dimensional module sends the depth of field information to the three-dimensional module. The three-dimensional module displays the depth of field area in a three-dimensional graphic in response to the depth of field information. For example, the inspector draws a circle on the display screen, and the area inside the circle belongs to the depth of field area to be displayed.

[0055] When the inspectors view the three-dimensional graphics and their attributes within the specified depth of field area, the image is clear. When the operator reaches the edge of the depth of field area, the three-dimensional graphics within the edge are high-definition, and the clarity of the three-dimensional graphics outside the edge is reduced, thereby prompting the operator to view content beyond the depth of field area, allowing the operator to re-plan the depth of field area to avoid viewing without a clear target. In particular, when the inspectors conduct real-time inspections in the factory area corresponding to the depth of field area, the information and images of the equipment obtained are clear. When the inspectors approach or walk out of the edge of the depth of field area, the interactive terminal sends a reminder message to remind the personnel that they have deviated from the inspection of the specified depth of field area.

[0056] Preferably, for equipment with abnormal properties, the two-dimensional graphic displays the two-dimensional graphic of the equipment that issues the alarm information, and at least one other equipment associated with the abnormal problem of the equipment, in a prominent color and / or logo. That is, the operator can clearly see the equipment that issues the alarm and the associated equipment related to the alarm on the two-dimensional view. The inspection personnel can view the three-dimensional graphics and their attribute information of the alarm equipment and its associated equipment in detail in a high-definition manner on the three-dimensional view by delineating the depth of field area containing the alarm equipment and the associated equipment on the two-dimensional view. That is, by highlighting the alarm equipment and its associated equipment, the inspection personnel can quickly determine the equipment that needs to be focused on, avoiding the drawbacks of aimlessly checking the equipment to find the cause of the alarm. When the inspection personnel inspect the alarm equipment in the factory area in real time, they can find the corresponding physical equipment based on the highlighted image display in the two-dimensional view or the three-dimensional view. In particular, after the inspectors use wearable devices to locate their real-time positions within the factory, the three-dimensional module prompts the inspectors with the relative position information of the equipment to be checked and the inspectors based on the inspectors' direction positioning. For example, the prompt information is "XX equipment is on your left / left front / right / right front / behind", etc., which makes it easy for inspectors to quickly lock on to the equipment that needs to be inspected.

[0057] If the method for defining the depth of field area is not set, when the inspectors view the equipment in the three-dimensional view on the interactive terminal, the three-dimensional module cannot determine the equipment that the inspectors are about to view next, and can only display all the equipment in the current three-dimensional view in high definition. However, in reality, there are many three-dimensional graphics and their attributes of equipment that the inspectors will not view in detail. This causes the three-dimensional module to send excessive and unnecessary data to the interactive terminal, resulting in data delays in the interactive terminal and slow updates of the three-dimensional graphics, affecting the screen update speed of the display device. The present invention specifies the depth of field area, and the three-dimensional module can reduce the amount of data transmitted to the interactive terminal for viewing. The three-dimensional module filters the high-definition data of the three-dimensional graphics and its attribute data based on the depth of field area sent by the two-dimensional module and the list of equipment in the depth of field area, and sends it to the interactive terminal. That is, the three-dimensional module can filter and transmit data in a targeted manner based on the depth of field area, thereby reducing the transmission of a large amount of unnecessary data.

[0058] Preferably, the depth of field area can be set not only in the two-dimensional view but also in the three-dimensional view. When the inspection personnel define the depth of field area in the three-dimensional view, the three-dimensional module can search for data according to the depth of field area and provide a list of the equipment in the depth of field area. The inspection personnel view the information they need according to the list.

[0059] Example 2

[0060] This embodiment is a further improvement on Embodiment 1, and the repeated contents will not be repeated here.

[0061] In this embodiment, the interactive terminal is preferably an augmented reality device. The device or system that uses augmented reality to control and visualize the controlled object in the prior art has the following defects: first, in the existing device that controls the visualized object through the augmented reality device, the inspector wearing the augmented reality device can only determine his own position based on the positioning information, but cannot confirm his relative position in the factory; second, the inspector wearing the augmented reality device cannot view the two-dimensional view and the three-dimensional view of the controlled object at the same time, nor can it display the depth of field of the three-dimensional view through the area setting of the two-dimensional view. In the prior art, when the inspector inspects the controlled object in the factory, he can only read the relevant information of the controlled object from the controlled objects and their three-dimensional graphic components around him, and inspect the controlled objects in turn according to a certain route. Most of the controlled objects will not have problems during the inspection. This inspection method will reduce the efficiency of the inspection and cannot detect the controlled objects with problems in advance. How to improve the inspection efficiency of the inspectors is an unresolved technical problem.

[0062] Based on the shortcomings of existing technologies, a system for real-time control and visualization of digital factories in an augmented reality environment is proposed. Figure 6As shown, it at least includes a server system 1, a control system 60, a plurality of controlled objects 70 and an augmented reality device 80. The control system 60 establishes a communication connection with the plurality of controlled objects 70 to obtain the attribute information and operating status of the controlled objects and control the controlled objects. The server system 1 establishes a communication connection with the control system 60 and the augmented reality device 80 respectively for data transmission. Among them, the server system 1 includes at least a two-dimensional module 10 and a three-dimensional module 20. The augmented reality device 80 establishes a communication connection with the two-dimensional module 10 and the three-dimensional module 20 respectively.

[0063] Preferably, the controlled object is assigned a value individually in units of components. The two-dimensional module 10 in the present invention is used to display the controlled object in a two-dimensional manner, such as Figure 8 Preferably, the patrol personnel can operate and display the two-dimensional graphics of any controlled object through the augmented reality device 80.

[0064] The three-dimensional module 20 of the present invention is used to display the controlled object in a three-dimensional manner, such as Figure 4 and 5 As shown, the inspection personnel can display the controlled object currently being controlled or viewed by the inspection personnel. Specifically, in the interactive module of at least one augmented reality device 80, the inspection personnel can operate to display the three-dimensional graphics of any controlled object.

[0065] Preferably, the properties of the controlled object can be displayed in parallel in two-dimensional and three-dimensional ways on the display module of the augmented reality device in response to the operation of the inspection personnel. Preferably, after the property data of the controlled object is changed by the operator and / or the inspection personnel, the relevant property parameters in the two-dimensional module and the three-dimensional module will change synchronously, so that the two-dimensional module and the three-dimensional module always display the controlled object in a real-time display manner with consistent property data. Compared with the augmented reality device in the prior art that can only display a two-dimensional view or a three-dimensional view, the parallel working mode of the two-dimensional module and the three-dimensional module of the present invention does not occupy the control network bandwidth, has a small workload, reduces the data delay display phenomenon of the three-dimensional model server and the two-dimensional module, and significantly improves the working efficiency of the graphic display, thereby realizing the synchronous real-time display of two-dimensional graphics and three-dimensional graphics.

[0066] For example, when the inspectors wear the augmented reality device 80 and walk in the factory area, they can view the real-time data of different controlled objects. Among them, the two-dimensional view and the three-dimensional view of the factory are displayed in parallel, that is, they are displayed on the display module at the same time, and adaptive changes are made at the same time. For example, when the inspectors view the controlled objects and their properties in the three-dimensional view, the two-dimensional view displayed by the display module can display the two-dimensional controlled objects in the area where the inspectors are located and the real-time movement trajectory of the inspectors in real time. The inspectors can set the display of the two-dimensional view and the three-dimensional view as needed, such as selecting the display area, the controlled object, the related information of the controlled object, etc., through the display module with interactive functions of augmented reality. For example, the two-dimensional view displays the two-dimensional controlled objects and the trajectory information of the inspectors in parallel at the angular position on the display screen of the three-dimensional view, or the two-dimensional view and the three-dimensional view are displayed separately in parallel screens.

[0067] Preferably, a mapping relationship is established between the two-dimensional graphic component of the controlled object and the three-dimensional graphic component of the controlled object, so that the inspection personnel can locate the two-dimensional graphic component in the three-dimensional view, and the inspection personnel can select the controlled object displayed in the three-dimensional view in the two-dimensional view. When the inspection personnel clicks on a component on the display of the two-dimensional module, the secondary module responds to the click of the two-dimensional graphic component of the controlled object, and displays the three-dimensional graphic component corresponding to the two-dimensional graphic component of the controlled object according to the pre-setting of the two-dimensional graphic component by the operator. The present invention enables the operator to conveniently switch between two-dimensional graphics and three-dimensional graphics through the setting of mapping association. Preferably, the inspection personnel determine the depth of field displayed in the three-dimensional view by clicking on the area of ​​the two-dimensional view of the augmented reality device 80. Such a setting is conducive to the inspection personnel to quickly determine the inspection area and the controlled object to be inspected, so as to conduct inspections in a corresponding inspection manner.

[0068] In real factories, the factory covers a large area, and the inspectors are prone to aimless and confused inspections during the inspection process, that is, they are not clear about which controlled objects need to be inspected and which areas should be inspected first. Even if the inspection is carried out according to the factory's preset inspection process, the inspection efficiency will be reduced due to physical problems. If the factory has enough inspectors, the personnel cost will increase accordingly, and the inspection efficiency will be limited. Increasing the number of inspectors alone cannot significantly improve the inspection efficiency.

[0069] The present invention takes advantage of the parallel display of two-dimensional modules and three-dimensional modules, allowing inspectors to combine online inspections with offline inspections, which saves physical effort and enables real inspections of controlled objects. In response to the depth of field area defined based on the two-dimensional view, the three-dimensional module displays a three-dimensional graphic component of at least one controlled object in the depth of field area. For example, the inspector defines the depth of field area of ​​the three-dimensional view with an undefined geometric shape on the displayed two-dimensional view, and the two-dimensional module sends the depth of field information to the three-dimensional module. The three-dimensional module displays the depth of field area in a three-dimensional graphic in response to the depth of field information. The inspectors conduct online inspections on the controlled objects in the three-dimensional view one by one. Among them, the inspectors can conduct a first check on the controlled objects in the depth of field area and determine at least one controlled object that needs to be viewed at a close distance.

[0070] The area delimited in the two-dimensional view can be a specified depth of field range. The depth of field range contains several controlled objects. When the inspection personnel delimit the depth of field area, the three-dimensional view displays the three-dimensional graphic components of the controlled object according to the specified depth of field range, and adjusts the clarity of the controlled object with the operation of the inspection personnel, so that the clarity of the three-dimensional graphic components of the controlled object changes based on the needs of the operation. In this way, the inspection personnel can inspect the controlled objects in a planned manner.

[0071] Preferably, the three-dimensional view of the present invention requests data according to the depth range specified in the two-dimensional scene, and provides a three-dimensional view containing the controlled object and the equipment associated with the controlled object within the depth range.

[0072] Preferably, the present invention preliminarily displays the three-dimensional graphics of the controlled object at a preset angle. Among them, in response to the operator's click information or display angle request, the three-dimensional module adjusts the display angle of the controlled object to the specified angle. Such display is conducive to the operator to view the operation of each angle of the controlled object and avoid the limitation of the viewing angle. Preferably, the three-dimensional module 20 displays the behavior trajectory of the task in the three-dimensional view based on the positioning information of the augmented reality device. The patrol personnel can view the relative position relationship between the three-dimensional character representing themselves and the controlled object in the three-dimensional view, so that they can quickly find the controlled objects that need to be inspected around themselves based on the relative position relationship, and can quickly improve the inspection efficiency. When the two-dimensional module and the three-dimensional module are in parallel, the two-dimensional view synchronously displays the position information of the patrol personnel in a two-dimensional manner, which is conducive to the patrol personnel to know the area they are patrolling, and based on the corresponding three-dimensional view that needs to be displayed through the area adjustment of the two-dimensional graphics, it is more conducive to the patrol personnel to plan a scientific inspection route through the two-dimensional view.

[0073] Preferably, when the inspectors inspect the controlled objects of the specified type, the two-dimensional module displays all the locations of the controlled objects of the specified type and displays at least one inspection route in response to the inspection personnel's point selection operation; the three-dimensional module displays the inspection scene corresponding to the location information in response to the location information of the inspectors, and gives a reminder when the inspectors reach the specified range of the controlled objects of the specified type, so as to avoid the inspectors from omitting or missing the controlled objects to be inspected. Such a setting enables the inspectors to conduct inspections in a focused manner during the inspection process, without turning back due to missing the inspection target, and without generating psychological pressure due to unclear relative distances between themselves and the inspection target. The inspectors can clearly understand the regional location of their own factory through the two-dimensional view, and determine the information of the controlled objects around their own location through the three-dimensional view.

[0074] In the prior art, the inspectors determine the controlled object and retrieve the properties of the controlled object and related information through the image information, coding information, QR code information, and voice information of the controlled object collected by the augmented reality device 80. This method of the prior art requires the inspectors to first contact the controlled object and then inspect the controlled object. In the case where the coding and QR code of the controlled object are not convenient to collect, it is difficult for the inspectors to determine the controlled object and retrieve related information. Especially for large-volume equipment, large pen-related graphics and even large printed coding information are not conducive to the collection of augmented reality devices. Smaller coding information requires the inspectors to reach the vicinity of the coding plate under the large equipment to collect it, which will undoubtedly increase the workload of the inspectors, cause unnecessary turns in the inspectors' paths, and reduce the efficiency of inspections.

[0075] In the present invention, both the two-dimensional view displayed by the two-dimensional module and the three-dimensional view displayed by the three-dimensional module can display the real-time position information of the augmented reality device in real time. When the inspection personnel need to retrieve relevant information about the controlled object around them, the inspection personnel can select the controlled object and obtain its relevant information based on the relative position of the equipment in the two-dimensional view and the characteristics of the three-dimensional graphic components corresponding to the controlled object in the three-dimensional view, without the need for the augmented reality device to determine the controlled object by collecting the information of the controlled object.

[0076] Preferably, the augmented reality device can determine the controlled object in the corresponding direction of the position of the inspector based on the action characteristics of the inspector, and display the relevant information of the controlled object based on the operation of the inspector. During the inspection process of the inspector, the inspector's forward direction is determined, and the direction of the controlled objects around the inspector's position relative to the inspector is determined. The augmented reality device displays at least one controlled object within a certain range of the designated direction of the inspector based on the operation of tilting and / or shaking in the designated direction. For example, the augmented reality device displays the properties of the closest controlled object on the right side of the inspector and its related information based on the operation of tilting and / or shaking in the right direction when the inspector wears the augmented reality device. Similarly, the inspector wears the augmented reality device and tilts and / or shakes in the left direction to display the properties of the closest controlled object on the left side of the inspector and its related information. In this way, the inspector can determine the controlled object and check the operating status of the controlled object without approaching the controlled object. Generally speaking, the operating status of most controlled objects is normal, which can save 80% of the working time and walking path of the inspector. When it is necessary to get close to the controlled object, the inspector will get close to the controlled object for detailed equipment inspection. For example, if the inspector checks the maintenance information of the controlled object and finds that the equipment was repaired and the visible parts were replaced the day before, the inspector needs to get close to the controlled object to check whether the repaired visible parts are consistent with the recorded information.

[0077] The action features set by the augmented reality device of the present invention are not limited to the listed left or right tilting actions, but can also be other action features, as long as the augmented reality device can recognize the action features and determine the controlled object.

[0078] Preferably, the two-dimensional module and the three-dimensional module of the present invention also store the real-time data and historical data of the controlled object. The real-time data and historical data come from the control system. An I / O module is provided between the control system and the controlled object. That is, the I / O module establishes a communication connection with the controlled object. For example, a temperature transmitter is provided in the controlled object. When the temperature in the equipment rises, the output current signal (4-20mA) of the transmitter increases, and the current signal is transmitted to the I / O module in the control room through a cable. After analog / digital (A / D) conversion, the data is uploaded to the control system for each operation station and each server to call.

[0079] Preferably, the control system transmits the real-time data to the two-dimensional module and the three-dimensional module in parallel. Preferably, the server system also includes a history server. The control system sends the real-time data to the history server for transfer and storage to form time-related history data.

[0080] Preferably, Figure 7As shown, the system for real-time control and visualization of digital factories under an augmented reality environment of the present invention also includes a two-dimensional operator station 81 and a three-dimensional operator station 82. Both the two-dimensional operator station 81 and the three-dimensional operator station 82 establish communication connections with the server system 1. The two-dimensional operator station 81 is used to display a two-dimensional graphic of a controlled object at at least one operator station. The three-dimensional operator station 82 is used to display a two-dimensional graphic of a controlled object at at least one operator station. The two-dimensional operator station 81 and the three-dimensional operator station 82 can be operated by the factory manager to display two-dimensional views and three-dimensional views to view relevant information of the controlled object. When the inspection personnel modify the properties of the controlled object through the two-dimensional operator station 81 or the three-dimensional operator station 82, the properties of the controlled object displayed by the two-dimensional operator station 81, the three-dimensional operator station 82 and the augmented reality device 80 change synchronously, which is conducive to the inspection personnel to conduct inspections based on the latest information during the inspection process and reduce the error rate of the inspection.

[0081] It should be noted that the above-mentioned specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention specification and its drawings are illustrative and do not constitute a limitation on the claims. The scope of protection of the present invention is defined by the claims and their equivalents. The present invention specification contains multiple inventive concepts, such as "preferably", "according to a preferred embodiment" or "optionally", all of which indicate that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application based on each inventive concept.

Claims

1. A virtual inspection system, comprising at least a server system, at least one interactive terminal and at least one control system, wherein the server system establishes communication connections with the at least one interactive terminal and the at least one control system respectively, characterized in that: The server system at least includes a two-dimensional module for displaying a two-dimensional graphic component of a controlled object and a three-dimensional module for displaying a three-dimensional graphic component of a controlled object. The attributes of the controlled object can be displayed in parallel on the interactive terminal in two-dimensional and three-dimensional ways in response to the operation of the inspection personnel. A mapping relationship is established between the two-dimensional graphic component of the controlled object and the three-dimensional graphic component. In response to the inspection personnel's operation on the two-dimensional view at the interactive terminal, the three-dimensional module can locate the three-dimensional graphic component corresponding to the specified two-dimensional graphic component in the three-dimensional view. When the inspection personnel define the depth of field area, the 3D view will display the 3D graphic components of the controlled object according to the specified depth of field area. When the inspection range of the inspection personnel approaches or goes out of the edge of the depth of field area, the interactive terminal sends a reminder message to remind the personnel that the inspection range deviates from the inspection range of the specified depth of field area.

2. The virtual inspection system according to claim 1, characterized in that: The two-dimensional module and the three-dimensional module simultaneously change the two-dimensional graphic component and the three-dimensional graphic component of the controlled object based on the property change of the controlled object.

3. The virtual inspection system according to claim 1 or 2, characterized in that: In response to the inspection personnel specifying the depth of field area in the two-dimensional view, the three-dimensional module sends the data of the three-dimensional graphics of the equipment located in the depth of field area to the interactive terminal. The three-dimensional view is made to display the three-dimensional graphics of the equipment located in the depth of field area in a clear manner, and the three-dimensional view is made to display the three-dimensional graphics of the equipment located outside the depth of field area in an unclear manner.

4. The virtual inspection system according to claim 1, characterized in that: The server system also includes a graphics server, which is used to store attribute data, digital engineering data and graphic engineering data of the controlled object sent by the control system. The graphic server fuses the real-time attribute data of the controlled object with the graphic data to form a three-dimensional graphic component capable of three-dimensional display, and displays the attributes of the controlled object in a three-dimensional view.

5. The virtual inspection system according to claim 1, characterized in that: In response to the interactive operation of the inspection personnel on the interactive terminal, the two-dimensional view of the controlled object jumps to the corresponding three-dimensional view.

6. The virtual inspection system according to claim 3, characterized in that: Inspection personnel can delineate the depth of field area containing the alarm device and related equipment on the two-dimensional view of the interactive terminal, and view the three-dimensional graphics and attribute information of the alarm device and its related equipment in detail in high definition on the three-dimensional view of the interactive terminal.

7. A virtual inspection method, characterized in that: The method at least comprises: Establish a mapping relationship between the two-dimensional graphic components and the three-dimensional graphic components of the controlled object. In response to the inspection personnel's operation on the two-dimensional view at the interactive terminal, the three-dimensional module can locate the three-dimensional graphic component corresponding to the specified two-dimensional graphic component in the three-dimensional view. In response to the inspection personnel's operation on the controlled object at the interactive terminal, the properties of the controlled object can be displayed in two-dimensional and three-dimensional ways in parallel at the interactive terminal. When the inspection personnel define the depth of field area, the 3D view will display the 3D graphic components of the controlled object according to the specified depth of field area. When the inspection range of the inspection personnel approaches or goes out of the edge of the depth of field area, the interactive terminal sends a reminder message to remind the personnel that the inspection range deviates from the inspection range of the specified depth of field area.

8. The virtual inspection method according to claim 7, characterized in that: The method further includes: the two-dimensional module and the three-dimensional module simultaneously change the two-dimensional graphic component and the three-dimensional graphic component of the controlled object based on the property change of the controlled object.

9. The virtual inspection method according to claim 7 or 8, characterized in that: The method further comprises: In response to the inspection personnel specifying the depth of field area in the two-dimensional view, the interactive terminal receives and displays the three-dimensional graphics of the equipment located in the depth of field area sent by the three-dimensional module, wherein: The 3D view displays the 3D graphics of the equipment located in the depth of field area in a clear manner, and the 3D view displays the 3D graphics of the equipment located outside the depth of field area in an unclear manner.

10. A visual factory system in an augmented reality environment, characterized in that: At least including an augmented reality device, a server system and several controlled objects, The server system comprises at least a two-dimensional module and a three-dimensional module, wherein the two-dimensional module displays the two-dimensional graphic components of the controlled object in a two-dimensional manner; The three-dimensional module displays the three-dimensional graphic components of the controlled object in a three-dimensional manner; The attribute data of the controlled object can be displayed in parallel in the form of a two-dimensional graphic and a three-dimensional view in response to the operation instruction of the augmented reality device, The two-dimensional module and / or the three-dimensional module can control the controlled object based on the operation instruction fed back by the augmented reality device, When the inspection personnel define the depth of field area, the 3D view will display the 3D graphic components of the controlled object according to the specified depth of field area. When the inspection range viewed by the inspection personnel approaches or goes out of the edge of the depth of field area, the augmented reality device sends a reminder message to remind the personnel that the inspection range deviates from the inspection range of the specified depth of field area.

11. The visual factory system in an augmented reality environment according to claim 10, characterized in that: In response to a definition operation of a depth of field region of a two-dimensional view of a controlled object, the three-dimensional module sends and displays a three-dimensional graphic component of the controlled object within the depth of field region to the augmented reality device.

12. The visual factory system in an augmented reality environment according to claim 10 or 11, characterized in that: The three-dimensional graphics component is formed by the graphics server fusing the graphics data with the attribute data, and is sent to the three-dimensional module in the form of a picture frequency stream. The three-dimensional module displays the received three-dimensional graphic component in a rendering manner.

13. The visual factory system in an augmented reality environment according to claim 10, characterized in that: In response to the operation information of the action characteristics of the augmented reality device, the three-dimensional module determines the controlled object that needs to be viewed and / or controlled based on the inspection personnel's forward direction and action characteristics.

Images