A production equipment monitoring method, device and electronic equipment based on AR glasses

Abstract

A production equipment monitoring method, device, and electronic device based on AR glasses relate to the field of augmented reality. In response to a user's monitoring operation on a first production equipment, the first production equipment is obtained; a first digital twin model corresponding to the first production equipment is obtained from a preset digital twin model library; the first digital twin model is used to simulate a first operating state of the first production equipment to obtain first primary operating state information corresponding to the first production equipment in real time; the first primary operating state information is displayed to the user in a preset manner; a first target instruction sent by the user is obtained to obtain a first viewing intention; secondary operating state information is obtained based on the first viewing intention; the secondary operating state information is displayed to the user in a preset display manner, so that the user can monitor the first production equipment based on the secondary operating state information. This solves the problem that when a user wishes to view other types of operating data, AR glasses cannot display according to the user's viewing needs.

Description

Technical Field

[0001] The present application relates to the field of augmented reality, and in particular to a production equipment monitoring method, device, and electronic device based on AR glasses. Background Art

[0002] With the advancement of industrial automation and intelligent manufacturing, augmented reality (AR) technology is increasingly being used in production equipment monitoring. Through AR glasses, users can intuitively view the operating status, performance indicators, and potential maintenance alerts of production equipment, significantly improving monitoring efficiency and facilitating equipment management.

[0003] However, when monitoring the operating status of production equipment through AR glasses, only fixed data types will be displayed in the AR glasses. When users want to view other types of operating data, the AR glasses cannot display according to the user's viewing needs.

[0004] Therefore, there is an urgent need for a production equipment monitoring method, device and electronic equipment based on AR glasses. Summary of the Invention

[0005] The present application provides a production equipment monitoring method, device and electronic device based on AR glasses, which solves the problem that when the user wants to view other types of operating data, AR glasses cannot display according to the user's viewing needs.

[0006] In the first aspect of the present application, a production equipment monitoring method based on AR glasses is provided, the method comprising: in response to a user's monitoring operation on a first production equipment, obtaining the first production equipment, where the first production equipment is a production equipment in the user's field of view; obtaining a first digital twin model corresponding to the first production equipment in a preset digital twin model library, where the preset digital twin model library stores the correspondence between the first production equipment and the first digital twin model; the first digital twin model is used to simulate a first operating state of the first production equipment to obtain first main operating state information corresponding to the first production equipment in real time; displaying the first main operating state information to the user in a preset manner; obtaining a first target instruction sent by the user; the first target instruction is used to represent the user's first viewing intention; the first viewing intention is to view the secondary operating state information corresponding to the first production equipment; obtaining the secondary operating state information according to the first viewing intention; displaying the secondary operating state information to the user in a preset display manner, so that the user can monitor the first production equipment based on the secondary operating state information.

[0007] By adopting the above technical solution, a digital twin model corresponding to each production device is constructed, and the operating status information corresponding to each device is obtained based on the digital twin model. When a user monitors the first production device, the first primary operating status information corresponding to the first production device is displayed to the user. Furthermore, if the user does not wish to view the first primary operating status information, the user's first target instruction is obtained, and the user's viewing intention is determined based on the first target instruction. The secondary operating status information that the user desires to view is then obtained based on the viewing intention, and the secondary operating status information is displayed to the user in place of the first primary operating status information. This solves the problem of AR glasses being unable to display data according to the user's viewing needs when the user desires to view other types of operating data.

[0008] Optionally, obtaining the first production equipment specifically includes: obtaining a target image of the target production equipment, the target production equipment being the production equipment in the user's field of view determined by eye tracking technology; obtaining a target feature in the target image, the target feature being the feature corresponding to the target production equipment; performing similarity calculation between the target feature and the feature group corresponding to the first production equipment in a preset feature library to obtain a target similarity value, the preset feature library storing the first production equipment, the feature group, and the correspondence between the first production equipment and the feature group; judging whether the target similarity value is greater than a preset similarity; if the target similarity is greater than the preset similarity, confirming that the target production equipment is the first production equipment, and obtaining the first production equipment.

[0009] By adopting the above technical solution, by obtaining the target image of the target production equipment and obtaining the target features in the target image, the target features are calculated to obtain a target similarity value by similarity calculation with the feature group corresponding to the first production equipment in the preset feature library, and then by judging whether the target similarity value is greater than the preset similarity, when the target similarity value is greater than the preset similarity, it is confirmed that the target production equipment monitored by the user is the first production equipment.

[0010] Optionally, before obtaining the first digital twin model corresponding to the first production equipment in the preset digital twin model library, the method also includes constructing a preset digital twin model library, specifically including: obtaining basic operating information corresponding to the first production equipment, the basic operating information including physical characteristic information, performance parameter information, production capacity information and working status information; constructing the first digital twin model corresponding to the first production equipment based on the basic operating information; and saving the first production equipment, the first digital twin model and the correspondence between the first production equipment and the first digital twin model in the preset digital twin model library.

[0011] By adopting the above technical solution, a numerical twin model is constructed for each production equipment, wherein the first digital twin model corresponds to the first production equipment, and each production equipment, each digital twin model, and the correspondence between each production equipment and each first digital twin model are saved in a preset digital twin model library, so that after identifying that the target production equipment monitored by the user is the first production equipment, the first digital twin model corresponding to the first production equipment is obtained through the correspondence, and the operating status information of the first production equipment is obtained based on the operation simulation of the first operating state of the first production equipment by the first digital twin model.

[0012] Optionally, after displaying the main operating status information to the user, the method also includes: obtaining a second target instruction sent by the user, the second target instruction is used to represent the user's second viewing intention, the second viewing intention is to simultaneously view the second main operating status information corresponding to the second production device, and the second production device is a production device in the user's field of view; obtaining the second main operating status information according to the second viewing intention; displaying the first main operating status information and the second main operating status information to the user in a comparative display manner, so that the user can compare and monitor the first production device and the second production device based on the first main operating status information and the second main operating status information.

[0013] By adopting the above technical solution, when the user wants to simultaneously monitor the second main operating status information of the second production equipment in order to compare the performance of the two equipment, the user can use the second target instruction to obtain the second main operating status information corresponding to the second production equipment, and display the first main operating status information and the second main operating status information to the user in a comparative display manner, so that the user can intuitively see the performance of the two equipment, and thus make timely analysis and decisions.

[0014] Optionally, obtaining the second main operating status information specifically includes: obtaining the second digital twin model corresponding to the second production equipment in a preset digital twin model library, the preset digital twin model library stores the second production equipment, the second digital twin model and the correspondence between the second production equipment and the second digital twin model, and the second digital twin model is used to simulate the second operating status of the second production equipment; obtaining the second main operating status information corresponding to the second production equipment through the second digital twin model.

[0015] By adopting the above-mentioned technical solution, the second digital twin model corresponding to the second production equipment is obtained from the preset digital twin model library, so that when the user wants to simultaneously monitor the second main operating status information of the second production equipment, the second digital twin model corresponding to the second production equipment is obtained through the corresponding relationship, and the operating status information of the second production equipment is obtained based on the operation simulation of the first operating status of the second production equipment by the second digital twin model.

[0016] Optionally, before obtaining the first production equipment in response to the user's monitoring operation on the first production equipment, the method also includes: obtaining the user's corresponding identity authentication information in response to the user's monitoring operation on the first production equipment; determining whether the identity authentication information is the preset identity authentication information; if the identity authentication information is the preset identity authentication information, obtaining the first production equipment.

[0017] By adopting the above technical solution, by setting identity authentication information and judging whether the identity authentication information input by the user is the preset identity authentication information, the permission to monitor production equipment is opened to the user only when the identity authentication information input by the user is the preset identity authentication information, thereby achieving safer and more controllable production equipment monitoring.

[0018] Optionally, the first main operating status information is displayed to the user in a preset manner, specifically including: displaying the first main operating status information at a preset position, the preset position being consistent with the user's line of sight and the position of the first production equipment in the real field of view.

[0019] By adopting the above technical solution, the first main operating status information is displayed to the user at a preset position, and the preset position is set according to the user's line of sight and the position of the first production equipment in the real field of view, thereby providing the user with a more natural and intuitive information viewing experience.

[0020] In a second aspect of the present application, a production equipment monitoring device based on AR glasses is provided, the device including an acquisition module and a processing module, wherein:

[0021] An acquisition module is used to acquire the first production equipment in response to the user's monitoring operation on the first production equipment, where the first production equipment is the production equipment in the user's field of view; acquire the first digital twin model corresponding to the first production equipment in the preset digital twin model library, and the preset digital twin model library stores the correspondence between the first production equipment and the first digital twin model; the first digital twin model is used to simulate the first operating state of the first production equipment to obtain the first main operating state information corresponding to the first production equipment in real time; acquire the first target instruction sent by the user; the first target instruction is used to represent the user's first viewing intention; the first viewing intention is to view the secondary operating state information corresponding to the first production equipment; and acquire the secondary operating state information according to the first viewing intention.

[0022] The processing module is used to display the first main operating status information to the user in a preset manner; and to display the secondary operating status information to the user in a preset display manner, so that the user can monitor the first production equipment according to the secondary operating status information.

[0023] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs any of the methods described above.

[0024] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to perform any of the above methods.

[0025] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0026] 1. By constructing a digital twin model corresponding to each piece of production equipment and obtaining the operating status information corresponding to each piece of equipment based on the digital twin model, the first primary operating status information corresponding to the first piece of production equipment is displayed to the user when the user is monitoring the first piece of production equipment. Furthermore, if the user does not wish to view the first primary operating status information, the user's first target instruction is obtained, and the user's viewing intention is determined based on the first target instruction. The secondary operating status information that the user desires to view is then obtained based on the viewing intention, and the secondary operating status information is displayed to the user in place of the first primary operating status information. This solves the problem of AR glasses being unable to display the data according to the user's viewing needs when the user desires to view other types of operating data.

[0027] 2. By obtaining the target image of the target production equipment and obtaining the target features in the target image, the target features are calculated to obtain a target similarity value by similarity calculation with the feature group corresponding to the first production equipment in the preset feature library, and then by judging whether the target similarity value is greater than the preset similarity, when the target similarity value is greater than the preset similarity, it is confirmed that the target production equipment monitored by the user is the first production equipment.

[0028] 3. When the user wants to monitor the second main operating status information of the second production equipment at the same time in order to compare the performance of the two equipment, the user can use the second target instruction to obtain the second main operating status information corresponding to the second production equipment, and display the first main operating status information and the second main operating status information to the user in a comparative display manner, so that the user can intuitively see the performance of the two equipment and make timely analysis and decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 1 is a flow chart of a production equipment monitoring method based on AR glasses provided in an embodiment of the present application;

[0030] Figure 2This is a module diagram of a production equipment monitoring device based on AR glasses provided in an embodiment of the present application;

[0031] Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present application.

[0032] Explanation of the reference numerals: 21, acquisition module; 22, processing module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0034] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application, the singular expressions "a", "an", "said", "above", "the", and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more of the listed items.

[0035] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0036] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0037] With the advancement of industrial automation and intelligent manufacturing, augmented reality (AR) technology is increasingly being used in production equipment monitoring. Through AR glasses, users can intuitively view the operating status, performance indicators, and potential maintenance alerts of production equipment, significantly improving monitoring efficiency and facilitating equipment management.

[0038] However, when monitoring the operating status of production equipment through AR glasses, only fixed data types will be displayed in the AR glasses. When users want to view other types of operating data, the AR glasses cannot display according to the user's viewing needs.

[0039] Therefore, there is an urgent need for a production equipment monitoring method, device and electronic equipment based on AR glasses.

[0040] Please refer to Figure 1 , which shows a flow chart of a production equipment monitoring method based on AR glasses provided in an embodiment of the present application. The method is applied to AR glasses. The flow chart mainly includes the following steps: S101 to S106.

[0041] Step S101 : In response to a user's monitoring operation on a first production device, the first production device is acquired, where the first production device is a production device in the user's field of view.

[0042] Specifically, when a user inspects the operating status of production equipment in a factory, the user can monitor the operating status of each production equipment by wearing AR glasses, where the production equipment can be machinery, tools or equipment used to produce goods or provide services. All equipment in the factory needs to be recorded on the server. Depending on the type of factory, the production equipment recorded in the AR glasses will also be different. For example, for a processing factory, the production equipment includes but is not limited to: lathes, milling machines, drilling machines, grinders, etc.; for a packaging factory, the production equipment includes but is not limited to: filling machines, sealing machines, labeling machines, etc. When the user is monitoring the first production equipment, if the user wears AR glasses, it ensures that the first production is in the user's field of view, so that the AR glasses can also obtain the first production equipment.

[0043] In a possible implementation, step S101 further includes: in response to the user's monitoring operation on the first production device, obtaining the user's corresponding identity authentication information; determining whether the identity authentication information is the preset identity authentication information; if the identity authentication information is the preset identity authentication information, obtaining the first production device.

[0044] Specifically, a user login system is built in the server, and each user in the factory needs to register before using the AR device to monitor the production equipment. Only registered users can monitor the production equipment through the AR device. Before using the AR device, the user enters the corresponding identity authentication information. The AR device obtains the identity authentication information entered by the user and determines whether the identity authentication information is the information entered when the user registered, that is, the preset identity authentication information, to authenticate the user. Only when the user passes the identity authentication can the production equipment be monitored through the AR device. The methods of identity authentication include but are not limited to password authentication, voice authentication, pupil recognition authentication, etc. This embodiment does not limit the method of identity authentication. It should be noted that the identity authentication information only needs to be authenticated once to confirm the identity of the user, and the user does not need to obtain the user's identity information every time he monitors a production device.

[0045] In a possible embodiment, step S101 also includes: obtaining a target image of the target production equipment, the target production equipment is a production equipment in the user's field of view determined by eye tracking technology; obtaining a target feature in the target image, the target feature is a feature corresponding to the target production equipment; performing similarity calculation between the target feature and the feature group corresponding to the first production equipment in the preset feature library to obtain a target similarity value, the preset feature library stores the first production equipment, the feature group, and the correspondence between the first production equipment and the feature group; judging whether the target similarity value is greater than the preset similarity; if the target similarity is greater than the preset similarity, confirming that the target production equipment is the first production equipment, and obtaining the first production equipment.

[0046] Specifically, to ensure accurate and efficient monitoring, the AR glasses must identify and select a target production equipment in the user's field of view. This equipment must correspond to one of multiple production equipment available for monitoring, as recorded in a server. If multiple production equipment are within the user's field of view, the AR glasses use eye tracking technology to determine the target production equipment the user wishes to monitor, defining the equipment on which the user's gaze rests as the target production equipment. The AR glasses then capture an image of the target production equipment and extract multiple features associated with the target production equipment, known as target features, from the image. The AR glasses then calculate similarity between the extracted target features and the feature groups corresponding to each production equipment in a preset feature library. The first production equipment is a production equipment in the preset feature library, which stores the feature groups corresponding to each production equipment in the factory. The target similarity is calculated between the target features and the feature groups corresponding to the first production equipment, and a determination is made as to whether the target similarity is greater than a preset similarity. If the target similarity is greater than the preset similarity, the first production equipment is identified as the production equipment on which the user's gaze rests. If multiple target similarities are found to be greater than the preset similarity, the equipment corresponding to the largest similarity is selected as the production equipment corresponding to the target production equipment. The preset similarity value can be set by the user according to actual needs, for example, to 80%, 85%, 90%, etc. This embodiment does not limit the setting of the preset similarity value. It should be noted that the server needs to save each production equipment in the factory and the feature group corresponding to each production equipment in the preset feature library in advance. The feature group is one or more features of the production equipment. When the factory updates the equipment or adds new equipment, the new production equipment and the feature group corresponding to the new production equipment can also be saved in the preset feature library, so that when the user monitors the new production equipment, the AR glasses can identify the production equipment.

[0047] Step S102: Obtain the first digital twin model corresponding to the first production equipment in the preset digital twin model library. The preset digital twin model library stores the correspondence between the first production equipment and the first digital twin model. The first digital twin model is used to simulate the first operating state of the first production equipment to obtain the first main operating state information corresponding to the first production equipment in real time.

[0048] Specifically, the server needs to pre-build a digital twin model corresponding to each piece of production equipment in the factory and store each piece of production equipment and its corresponding digital twin model in the server's digital twin model library. The digital twin model simulates the operating status of the corresponding production equipment to obtain the operating status information of the production equipment in real time. Once the AR glasses confirm that the production equipment being monitored by the user is the first production equipment, they can obtain the first digital twin model through the correspondence between the first production equipment and the first digital twin model. Based on the first digital twin model, they can obtain the first main operating status information corresponding to the first production equipment.

[0049] In a possible implementation, step S102 also includes: before obtaining the first digital twin model corresponding to the first production equipment in the preset digital twin model library, the method also includes constructing a preset digital twin model library, specifically including: obtaining basic operating information corresponding to the first production equipment, the basic operating information including physical characteristic information, performance parameter information, production capacity information and working status information; constructing the first digital twin model corresponding to the first production equipment based on the basic operating information; and saving the first production equipment, the first digital twin model and the correspondence between the first production equipment and the first digital twin model in the preset digital twin model library.

[0050] Specifically, when building a digital twin model for each piece of production equipment in a factory, basic operating information corresponding to each piece of equipment is first obtained, including basic operating information for the first piece of production equipment. This basic operating information includes, but is not limited to, the equipment's physical characteristics, performance parameters, production capacity, and operating status. The digital twin model is constructed through the following steps: The server first collects necessary data from the physical equipment, including, but not limited to, physical characteristics (e.g., dimensions, shape, and material), performance parameters (e.g., speed, temperature range, and pressure level), production capacity (e.g., output per unit time, efficiency), and current operating status information (e.g., on / off status, fault information, and maintenance records). The server then standardizes and integrates the collected data to facilitate understanding and use by the digital twin model. This standardization and integration process includes, but is not limited to, data cleansing, formatting, and synchronization with existing databases. Using this collected data, the server constructs a virtual replica of each piece of production equipment in a digital environment using advanced modeling techniques, including, but not limited to, 3D modeling, system dynamics modeling, and data-driven behavioral modeling, thereby constructing the corresponding digital twin model.

[0051] Step S103: Display the first main operating status information to the user.

[0052] Specifically, the AR glasses are connected to the server wirelessly to synchronize the first main operating status information of the first production equipment generated by the first digital twin model. The first main operating status information is the information in all operating status information of the first production equipment, which can be set by the user. All operating status information includes but is not limited to the real-time working parameters, performance indicators, production efficiency, fault alarms, etc. of the equipment. The AR glasses can process the received first main operating status information and convert it into visual elements that can be understood by the user, including but not limited to charts, indicator lights, digital readouts or status indications of 3D models. The AR glasses place the first main operating status information at a preset position in the AR glasses lens. The preset position is consistent with the user's line of sight and the position of the first production equipment in the real field of view, so that the user can intuitively match the displayed status information with the corresponding equipment.

[0053] Step S104, obtaining a first target instruction sent by the user; the first target instruction is used to indicate the user's first viewing intention; the first viewing intention is to view the secondary operating status information corresponding to the first production equipment.

[0054] Specifically, the user can perform other monitoring operations through different target commands. The first target command is one of the different target commands. The user sends the first target command to the AR glasses, and the AR glasses recognize the user's first viewing intention through the first target command. The first target command includes but is not limited to a voice command, a gesture command, etc. This embodiment does not limit the method for sending the first command.

[0055] Step S105: Acquire secondary running status information according to the first viewing intention.

[0056] Specifically, by obtaining the first instruction, the AR glasses perform intention analysis on the first instruction, thereby obtaining the user's first viewing intention and obtaining secondary operating status information based on the result of the intention analysis.

[0057] For example, when the first instruction is a gesture instruction, the AR glasses capture the user's gesture instruction, and the user needs to perform corresponding gesture operations in the field of view in order for the AR glasses to capture the gesture instruction. Assume that at this time, three main operating status information about the first production equipment are displayed at the position corresponding to the first production equipment in the lens of the AR glasses, and are displayed from top to bottom in the user's real field of view. If the user does not want to view the first status information, the display of this information can be changed or turned off through gesture instructions. For example, the user can make a swipe to the right gesture, and the AR glasses capture this action through the built-in gesture recognition system. The system will analyze the captured gesture and understand that the user's intention is to view the secondary operating status information. Next, the AR glasses will adjust the display of information according to the user's intention. If the right-swiping gesture is to not view a certain status and to view other information, that is, the secondary operating status information, then the AR glasses will turn off the display of the first status information when the user swipes right on the first status information, and display the secondary operating status information; when the first instruction is a voice instruction, the AR glasses obtain the user's voice instruction through the voice recognition system, and analyze the voice instruction to perform intention analysis. Before performing voice analysis, it can determine whether to start the voice recognition system by identifying whether the user says a preset instruction. For example, the user needs to say "Hello, world" first. When the voice recognition system recognizes "Hello, world", it will analyze the user's next voice. Otherwise, the user's voice will not be analyzed. If the user does not wish to view the first status information, they can use voice commands to "close the first status information." The AR glasses will use the voice recognition system to detect the voice command "close the first status information," analyze the user's intent, and then close the first status information. If the user wishes to view secondary operating status information, they can use voice commands to "display xx operating status information." The AR glasses will use the voice recognition system to detect the voice command "close the first status information," analyze the user's intent, and then display xx operating status information for the user. xx operating status information is any type of secondary operating status information. For primary operating status information and secondary operating status information, primary operating status information is commonly used operating status information for production equipment, while secondary operating status information is less commonly used operating status information for production equipment. For example, for a lathe, primary operating status information may include operating speed information, tool position information, cutting depth information, workpiece temperature information, etc., while secondary operating status information may include lubricating oil temperature information, motor load information, coolant information, etc. Commonly used operating status information refers to the information that the AR glasses display first each time the user monitors the device. Infrequently used status information refers to the information that is not displayed first each time the user monitors the device and will only be displayed after the user performs corresponding operations.Both the frequently used running status information and the infrequently used status information can be set according to the user's intention. This embodiment does not limit the setting of the frequently used running status information and the infrequently used status information.

[0058] Step S106: Display the secondary operating status information to the user in a preset display manner, so that the user can monitor the first production equipment according to the secondary operating status information.

[0059] Specifically, the AR glasses are connected to the server wirelessly to synchronize the secondary operating status information of the first production equipment generated by the first digital twin model. The secondary operating status information is all the operating status information of the first production equipment except the primary operating status information, which can also be set by the user. The AR glasses can process the received secondary operating status information and convert it into visual elements that the user can understand, including but not limited to charts, indicator lights, digital readouts or status indications of 3D models. The AR glasses place the secondary operating status information at the preset position of the first primary operating status information in the AR glasses lens to display it to the user instead of the first primary operating status information, so that the user can monitor the first production equipment based on the secondary operating status information.

[0060] In a possible embodiment, it also includes: obtaining a second target instruction sent by the user, the second target instruction is used to represent the user's second viewing intention, the second viewing intention is to simultaneously view the second main operating status information corresponding to the second production equipment, and the second production equipment is the production equipment in the user's field of view; according to the second viewing intention, obtaining the second digital twin model corresponding to the second production equipment in the preset digital twin model library, the preset digital twin model library stores the second production equipment, the second digital twin model and the correspondence between the second production equipment and the second digital twin model, and the second digital twin model is used to simulate the second operating state of the second production equipment; obtaining the second main operating status information corresponding to the second production equipment through the second digital twin model; displaying the first main operating status information and the second main operating status information to the user in a comparative display manner, so that the user can compare and monitor the first production equipment and the second production equipment based on the first main operating status information and the second main operating status information.

[0061] Specifically, the user can perform other monitoring operations through different target instructions. The second target instruction is one of the different target instructions. The user sends the second target instruction to the AR glasses, and the AR glasses recognize the user's second viewing intention through the second target instruction. The second viewing intention is to simultaneously view the second main operating status information corresponding to the second production equipment. When the AR glasses obtain the second instruction, the second digital twin model corresponding to the second production equipment is obtained from the preset digital twin model library, and the second main operating status information corresponding to the second production equipment is obtained through the second digital twin model; the first main operating status information and the second main operating status information are displayed to the user in a comparative display manner, so that the user can compare and monitor the first production equipment and the second production equipment based on the first main operating status information and the second main operating status information.

[0062] For example, a user is checking device A, and AR glasses display three key operating status information, such as its temperature, output rate, and energy consumption. The user then wants to simultaneously monitor the status of another device, device B, to compare their performance. The user issues a second target instruction to the AR glasses, perhaps a voice command, such as "Show device B's key operating status." After receiving this instruction, the AR glasses understand the user's intent through voice recognition. They then retrieve the digital twin model corresponding to device B from a pre-defined digital twin model library and obtain device B's second key operating status information, such as its temperature, output rate, and energy consumption. The AR glasses then present the key operating status information of devices A and B to the user in a comparative display. This comparative display displays the first and second key operating status information side by side in the user's field of view, or displays a direct performance comparison in the form of a chart. This allows the user to intuitively see the performance of the two devices and make timely analysis and decisions.

[0063] This application adopts the above-mentioned method to construct a digital twin model corresponding to each production device, and obtains the operating status information corresponding to each device based on the digital twin model, so that when the user monitors the first production device, the first primary operating status information corresponding to the first production device is displayed to the user. When the user does not want to view the first primary operating status information, the application obtains the user's first target instruction and obtains the user's viewing intention based on the first target instruction, thereby obtaining the secondary operating status information that the user wants to view based on the viewing intention, and displays the secondary operating status information to the user instead of the first primary operating status information, thereby solving the problem that when the user wants to view other types of operating data, the AR glasses cannot display according to the user's viewing needs.

[0064] Please refer to Figure 2, which shows a production equipment monitoring device based on AR glasses provided in an embodiment of the present application. The device is AR glasses, and the AR glasses include an acquisition module 21 and a processing module 22, wherein,

[0065] The acquisition module 21 is used to obtain the first production equipment in response to the user's monitoring operation on the first production equipment, where the first production equipment is the production equipment in the user's field of view; obtain the first digital twin model corresponding to the first production equipment in the preset digital twin model library, and the preset digital twin model library stores the correspondence between the first production equipment and the first digital twin model; the first digital twin model is used to simulate the first operating state of the first production equipment to obtain the first main operating state information corresponding to the first production equipment in real time; obtain the first target instruction sent by the user; the first target instruction is used to represent the user's first viewing intention; the first viewing intention is to view the secondary operating state information corresponding to the first production equipment; and obtain the secondary operating state information according to the first viewing intention.

[0066] The processing module 22 is configured to display the first primary operating status information to the user in a preset manner; and to display the secondary operating status information to the user in a preset manner, so that the user can monitor the first production equipment according to the secondary operating status information.

[0067] In one possible embodiment, the acquisition module 21 is used to acquire the first production equipment, specifically including: acquiring a target image of the target production equipment, the target production equipment being the production equipment in the user's field of view determined by eye tracking technology; acquiring target features in the target image, the target features being the features corresponding to the target production equipment; performing similarity calculation between the target features and the feature group corresponding to the first production equipment in the preset feature library to obtain a target similarity value, the preset feature library storing the first production equipment, the feature group, and the correspondence between the first production equipment and the feature group; determining whether the target similarity value is greater than the preset similarity; if the target similarity is greater than the preset similarity, confirming that the target production equipment is the first production equipment, and acquiring the first production equipment.

[0068] In one possible embodiment, before the acquisition module 21 is used to obtain the first digital twin model corresponding to the first production equipment in the preset digital twin model library, the method also includes constructing a preset digital twin model library, specifically including: obtaining basic operating information corresponding to the first production equipment, the basic operating information including physical characteristic information, performance parameter information, production capacity information and working status information; constructing the first digital twin model corresponding to the first production equipment based on the basic operating information; and saving the first production equipment, the first digital twin model and the correspondence between the first production equipment and the first digital twin model in the preset digital twin model library.

[0069] In one possible embodiment, the acquisition module 21 is used to obtain a second target instruction sent by the user after displaying the main operating status information to the user, and the second target instruction is used to represent the user's second viewing intention, and the second viewing intention is to simultaneously view the second main operating status information corresponding to the second production equipment, and the second production equipment is the production equipment in the user's field of view; obtain the second main operating status information according to the second viewing intention; display the first main operating status information and the second main operating status information to the user in a comparative display manner, so that the user can compare and monitor the first production equipment and the second production equipment based on the first main operating status information and the second main operating status information.

[0070] In one possible implementation, the acquisition module 21 is used to obtain the second main operating status information, specifically including: obtaining the second digital twin model corresponding to the second production equipment in the preset digital twin model library, the preset digital twin model library stores the second production equipment, the second digital twin model and the correspondence between the second production equipment and the second digital twin model, and the second digital twin model is used to simulate the second operating state of the second production equipment; through the second digital twin model, obtain the second main operating status information corresponding to the second production equipment.

[0071] In one possible embodiment, the acquisition module 21 is used to obtain the identity authentication information corresponding to the user in response to the user's monitoring operation on the first production equipment before acquiring the first production equipment; determine whether the identity authentication information is the preset identity authentication information; if the identity authentication information is the preset identity authentication information, acquire the first production equipment.

[0072] In one possible embodiment, the processing module 22 is used to display the first main operating status information to the user in a preset manner, specifically including: displaying the first main operating status information through a preset position, and the preset position is consistent with the user's line of sight and the position of the first production equipment in the real field of view.

[0073] It should be noted that the above embodiments provide devices that implement their functions using only the division of the above functional modules as examples. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0074] This application also provides an electronic device. Figure 3 , Figure 33 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. The electronic device may include: at least one processor 301, at least one communication bus 302, a user interface 303, at least one network interface 304, and a memory 305.

[0075] The communication bus 302 is used to implement the connection and communication between these components.

[0076] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0077] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0078] The processor 301 may include one or more processing cores. The processor 301 utilizes various interfaces and circuits to connect various components within the AR glasses. It executes instructions, programs, code sets, or instruction sets stored in the memory 305, as well as accesses data stored in the memory 305, to perform various functions of the AR glasses and process data. Optionally, the processor 301 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 301 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and applications; the GPU is responsible for rendering and drawing content displayed on the display; and the modem handles wireless communications. It is understood that the modem may also be implemented independently of the processor 301 and implemented on a separate chip.

[0079] Among them, the memory 305 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may also optionally be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an AR glasses-based production equipment monitoring application.

[0080] exist Figure 3 In the electronic device shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the production equipment monitoring application based on AR glasses stored in the memory 305. When executed by one or more processors 301, the electronic device executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited to the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0081] The present application also provides a computer-readable storage medium storing instructions, which, when executed by one or more processors, enable an electronic device to execute one or more of the methods described in the above embodiments.

[0082] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic, such as the division of units, which is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0084] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0085] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for enabling a computer device (which can be a personal computer, AR glasses, or network equipment, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes various media that can store program code, such as USB flash drives, mobile hard drives, magnetic disks, or optical disks.

[0087] The foregoing description is merely an exemplary embodiment of the present disclosure and does not limit the scope of the present disclosure. In other words, any equivalent variations and modifications made based on the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the disclosure and practical experience.

[0088] This application is intended to cover any modifications, uses or adaptations disclosed in this application, which follow the general principles disclosed in this application and include common knowledge or customary technical means in the technical field not disclosed in this application.

Claims

1. A production equipment monitoring method based on AR glasses, characterized in that: The method comprises: In response to a user's monitoring operation on a first production device, acquiring the first production device, where the first production device is a production device in the user's field of view; Obtaining a first digital twin model corresponding to the first production equipment from a preset digital twin model library, wherein the preset digital twin model library stores a correspondence between the first production equipment and the first digital twin model; the first digital twin model is used to simulate a first operating state of the first production equipment to obtain first main operating state information corresponding to the first production equipment in real time; Displaying the first main operating status information to the user in a preset manner; Acquire a first target instruction sent by the user; the first target instruction is used to indicate the first viewing intention of the user; the first viewing intention is to view the secondary operating status information corresponding to the first production device; acquiring the secondary operating state information according to the first viewing intention; Displaying the secondary operating status information to the user in the preset manner, so that the user can monitor the first production equipment according to the secondary operating status information; Obtaining a second target instruction sent by the user, where the second target instruction is used to indicate a second viewing intention of the user, where the second viewing intention is to simultaneously view second primary operating status information corresponding to a second production device, where the second production device is a production device in the user's field of view; acquiring the second main operating state information according to the second viewing intention; The first main operating status information and the second main operating status information are displayed to the user in a comparative display manner, so that the user can comparatively monitor the first production equipment and the second production equipment based on the first main operating status information and the second main operating status information.

2. The method according to claim 1, characterized in that Acquiring the first production equipment specifically includes: Acquire a target image of a target production device, where the target production device is a production device determined by eye tracking technology in the user's field of view; Acquire a target feature in the target image, where the target feature is a feature corresponding to the target production equipment; Calculating the similarity between the target feature and a feature group corresponding to a first production device in a preset feature library to obtain a target similarity value, wherein the preset feature library stores the first production device, the feature group, and the corresponding relationship between the first production device and the feature group; Determining whether the target similarity value is greater than a preset similarity; If the target similarity is greater than the preset similarity, the target production equipment is confirmed to be the first production equipment, and the first production equipment is acquired.

3. The method according to claim 1, characterized in that Before obtaining the first digital twin model corresponding to the first production equipment from the preset digital twin model library, the method further includes constructing the preset digital twin model library, specifically including: Obtaining basic operating information corresponding to the first production equipment, the basic operating information including physical characteristic information, performance parameter information, production capacity information, and working status information; Constructing a first digital twin model corresponding to the first production equipment based on the basic operating information; The first production equipment, the first digital twin model, and the corresponding relationship between the first production equipment and the first digital twin model are saved in the preset digital twin model library.

4. The method according to claim 1, wherein The obtaining of the second main operating status information specifically includes: Obtaining a second digital twin model corresponding to the second production equipment from a preset digital twin model library, wherein the preset digital twin model library stores the second production equipment, the second digital twin model, and a correspondence between the second production equipment and the second digital twin model, wherein the second digital twin model is used to simulate a second operating state of the second production equipment; Through the second digital twin model, the second main operating status information corresponding to the second production equipment is obtained.

5. The method according to claim 1, wherein Before acquiring the first production equipment in response to the user's monitoring operation on the first production equipment, the method further includes: In response to a user's monitoring operation on the first production equipment, obtaining identity authentication information corresponding to the user; Determining whether the identity authentication information is preset identity authentication information; If the identity authentication information is the preset identity authentication information, the first production device is obtained.

6. The method according to claim 1, characterized in that The displaying of the first main operating status information to the user in a preset manner specifically includes: The first main operating status information is displayed through a preset position, and the preset position is consistent with the user's line of sight and the position of the first production equipment in the real field of view.

7. A production equipment monitoring device based on AR glasses, used to perform the method according to claim 1, characterized in that: The device comprises an acquisition module (21) and a processing module (22), wherein: The acquisition module (21) is used to obtain the first production equipment in response to the user's monitoring operation on the first production equipment, the first production equipment being the production equipment in the user's field of view; obtaining the first digital twin model corresponding to the first production equipment in a preset digital twin model library, the preset digital twin model library storing the first production equipment, the first digital twin model and the corresponding relationship between the first production equipment and the first digital twin model; the first digital twin model is used to simulate the first operating state of the first production equipment to obtain the first main operating state information corresponding to the first production equipment in real time; obtaining the first target instruction sent by the user; the first target instruction is used to represent the user's first viewing intention; the first viewing intention is to view the secondary operating state information corresponding to the first production equipment; and obtaining the secondary operating state information according to the first viewing intention; The processing module (22) is used to display the first main operating status information to the user in a preset manner; and to display the secondary operating status information to the user in the preset manner, so that the user can monitor the first production equipment according to the secondary operating status information.

8. An electronic device, characterized in that: The electronic device comprises a processor (301), a communication bus (302), a user interface (303), a network interface (304) and a memory (305), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 6 is performed.

Images