Data processing method and related device

Abstract

The invention discloses a data processing method and a related device, which can be applied to the fields of cloud technology, digital twinning, automatic driving, auxiliary driving, intelligent traffic, traffic simulation and the like. And rendering a virtual scene of the to-be-monitored scene according to the acquired scene model data, and determining an instantiated object of the to-be-monitored equipment according to the equipment identifier of the to-be-monitored equipment. And calculating a world transformation matrix according to the obtained pose data of the to-be-monitored equipment, and rendering the instantiated object in the virtual scene through the world transformation matrix. And calculating a local transformation matrix of the joints according to the motion state of the joints included in the to-be-monitored equipment at the current moment, and determining the motion state of the virtual joints in the instantiated object at the current moment and the motion state of the component model connected with the virtual joints at the current moment based on the local transformation matrix. The method is more real-time, twinning reconstruction is carried out by taking the part and the joint as units, the operation condition of the to-be-monitored equipment is accurately restored, and monitoring of the to-be-monitored equipment is facilitated.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data processing method and related devices. Background Art

[0002] With the development of industrial technology, many industries need to use a variety of equipment, such as Internet equipment in the computer room, robotic arms and trolleys on the production line, etc. Real-time monitoring of these devices is an extremely important part of the current industrial production and operation management process, so that possible problems with the equipment can be discovered in time through real-time monitoring, so as to perform regular maintenance and avoid equipment failure.

[0003] With the development of science and technology, the means of real-time monitoring have gradually evolved from dedicated personnel to intelligent and information-based monitoring. The information-based monitoring means provided by relevant technologies mainly involve three-dimensional modeling of the equipment in advance, virtual layout combined with site information, and then obtaining the final state of the equipment action by pulling the control instructions of the equipment. The three-dimensional simulation software processes the data inside the simulation software according to the starting state and the final state of the action, deduce the entire movement process, and then simulate the movement process, so as to monitor according to the simulation display results.

[0004] Since the simulation software itself is based on control instructions, the states during its operation are all deduced through the simulation software, but the intermediate process may not be consistent with the operation process of the real equipment, and the deduction process takes a certain amount of time, so it has a lag. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a data processing method and related devices, which perform twin reconstruction on the operation status of the monitored device based on the acquired posture data of the monitored device at the current moment and the motion status of the joints included in the monitored device at the current moment. Since the posture data and motion status can reflect the real-time operation status of the monitored device, the operation status of the monitored device can be accurately and in real time restored, which is real-time. In addition, the twin reconstruction process restores the motion status of the monitored device in units of the joints and components included in the monitored device. For the monitored device including movable parts, the motion status of the monitored device can be restored more realistically and accurately, which is beneficial to the monitoring of the monitored device.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a data processing method, the method comprising:

[0008] Acquire scene model data of a scene to be monitored, and acquire device status data of a device to be monitored in the scene to be monitored, wherein the device status data includes a device identification of the device to be monitored, position data of the device to be monitored at a current moment, and a motion state of a joint included in the device to be monitored at the current moment;

[0009] Rendering a virtual scene of the scene to be monitored according to the scene model data, and determining an instantiated object of the device to be monitored according to the device identification of the device to be monitored, wherein the instantiated object is generated by a first component description file, and the first component description file is used to indicate the components included in the device to be monitored and the joints connecting the components;

[0010] Calculate the world transformation matrix corresponding to the device to be monitored according to the posture data of the device to be monitored, and render the instantiated object in the virtual scene through the world transformation matrix;

[0011] According to the motion state of the joint included in the device to be monitored at the current moment, the local transformation matrix of the joint is calculated, and the motion state of the virtual joint corresponding to the joint in the instantiated object at the current moment is determined based on the local transformation matrix, and the motion state of the component model connected to the virtual joint in the instantiated object at the current moment is determined based on the local transformation matrix.

[0012] On the one hand, an embodiment of the present application provides a data processing method, the method comprising:

[0013] Acquire real-time data of the device to be monitored collected at the data collection time, and acquire a first component description file of the device to be monitored;

[0014] Performing format conversion on the device real-time data based on the first component description file to obtain initial state data;

[0015] Determine device status data of the device to be monitored according to the initial status data, wherein the device status data includes a device identification of the device to be monitored, position data of the device to be monitored at a current moment, and a motion state of a joint included in the device to be monitored at the current moment;

[0016] Send the device status data of the device to be monitored to the device display terminal.

[0017] On the one hand, an embodiment of the present application provides a data processing device, the device comprising an acquisition unit, a rendering unit, a determination unit and a calculation unit:

[0018] The acquisition unit is used to acquire scene model data of the scene to be monitored, and to acquire device status data of the device to be monitored in the scene to be monitored, wherein the device status data includes a device identification of the device to be monitored, position data of the device to be monitored at a current moment, and a motion state of a joint included in the device to be monitored at the current moment;

[0019] The rendering unit is used to render a virtual scene of the scene to be monitored according to the scene model data;

[0020] The determining unit is used to determine the instantiation object of the device to be monitored according to the device identification of the device to be monitored, wherein the instantiation object is generated by a first component description file, and the first component description file is used to indicate the components included in the device to be monitored and the joints connecting the components;

[0021] The calculation unit is used to calculate the world transformation matrix corresponding to the device to be monitored according to the posture data of the device to be monitored;

[0022] The rendering unit is further used to render the instantiated object in the virtual scene through the world transformation matrix;

[0023] The calculation unit is further used to calculate the local transformation matrix of the joint according to the motion state of the joint included in the device to be monitored at the current moment;

[0024] The determination unit is also used to determine the motion state of the virtual joint corresponding to the joint in the instantiated object at the current moment based on the local transformation matrix, and to determine the motion state of the component model connected to the virtual joint in the instantiated object at the current moment based on the local transformation matrix.

[0025] On the one hand, an embodiment of the present application provides a data processing device, the device comprising an acquisition unit, a conversion unit, a determination unit and a sending unit:

[0026] The acquisition unit is used to acquire the real-time data of the device to be monitored collected at the data collection time, and to acquire the first component description file of the device to be monitored;

[0027] The conversion unit is used to convert the format of the device real-time data based on the first component description file to obtain initial state data;

[0028] The determining unit is used to determine the device state data of the device to be monitored according to the initial state data, wherein the device state data includes the device identification of the device to be monitored, the position and posture data of the device to be monitored at a current moment, and the motion state of the joints included in the device to be monitored at the current moment;

[0029] The sending unit is used to send the device status data of the device to be monitored to the device display terminal.

[0030] In one aspect, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:

[0031] The memory is used to store a computer program and transmit the computer program to the processor;

[0032] The processor is configured to execute the method described in any one of the preceding aspects according to instructions in the computer program.

[0033] On the one hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the processor executes the method described in any of the aforementioned aspects.

[0034] In one aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any of the aforementioned aspects when executed by a processor.

[0035] It can be seen from the above technical solution that when it is necessary to perform visual monitoring on the monitored device, the operation of the monitored device can be twinned and reconstructed. Specifically, the scene model data of the monitored scene can be obtained, and the device status data of the monitored device in the monitored scene can be obtained. The device status data includes the device identification of the monitored device, the posture data of the monitored device at the current moment, and the motion state of the joints included in the monitored device at the current moment. The posture data and the motion state can reflect the real-time operation of the monitored device. Then, the virtual scene of the monitored scene is rendered according to the scene model data, and the instantiated object of the monitored device is determined according to the device identification of the monitored device. The instantiated object is generated by a first component description file, and the first component description file is used to indicate the components included in the monitored device and the joints connecting the components. The instantiated object is an object that can be displayed, and after rendering, it looks like the body of the monitored device. In order to truly restore the operation of the monitored device in the monitored scene in the virtual scene, the world transformation matrix corresponding to the monitored device can be calculated according to the posture data of the monitored device, and the instantiated object is rendered in the virtual scene through the world transformation matrix, thereby restoring the posture of the monitored device in the monitored scene. Then, according to the motion state of the joints included in the device to be monitored at the current moment, the local transformation matrix of the joints is calculated, and the motion state of the virtual joints corresponding to the joints in the instantiated object at the current moment is determined based on the local transformation matrix, and the motion state of the component model connected to the virtual joints in the instantiated object at the current moment is determined based on the local transformation matrix, so as to restore the operation of the device to be monitored in the monitored scene through the operation of the instantiated object in the virtual scene, so as to perform visual monitoring of the device to be monitored based on the operation of the instantiated object in the virtual scene. Based on the acquired posture data of the device to be monitored at the current moment and the motion state of the joints included in the device to be monitored at the current moment, the operation of the device to be monitored is twin-reconstructed. Since the posture data and motion state can reflect the real-time operation of the device to be monitored, that is, the data used to control the instantiated object comes from the real-time operation of the device to be monitored, no deduction is required, so the operation of the device to be monitored can be accurately and in real time restored, which is real-time. Moreover, the twin reconstruction process restores the motion state of the device to be monitored based on the joints and components included in the device to be monitored. For the device to be monitored that includes movable parts, the motion state of the device to be monitored can be restored more realistically and accurately, which is conducive to the monitoring of the device to be monitored. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technical members in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 An application scenario architecture diagram of a data processing method provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0039] Figure 3 An example diagram of components obtained by disassembling a mechanical arm structure provided in an embodiment of the present application;

[0040] Figure 4 An example diagram of joints obtained by disassembling a mechanical arm structure provided in an embodiment of the present application;

[0041] Figure 5 An example diagram of a second component description file provided in an embodiment of the present application;

[0042] Figure 6 A flowchart of another data processing method provided in an embodiment of the present application;

[0043] Figure 7 An example diagram of the overall process of a data processing method provided in an embodiment of the present application;

[0044] Figure 8 A structural diagram of a data processing device provided in an embodiment of the present application;

[0045] Fig. 9 A structural diagram of another data processing device provided in an embodiment of the present application;

[0046] Fig.10 A structural diagram of a terminal provided in an embodiment of the present application;

[0047] Fig.11 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0049] In the context of actively building smart industries, concepts such as digital workshops, smart factories, and smart ports have been proposed one after another. In the industries represented by these concepts, there is a common demand, which is to monitor the operation of equipment in the field in real time. With the development of science and technology, the means of real-time monitoring have gradually developed from dedicated personnel to intelligent and information-based.

[0050] The information monitoring means provided by related technologies mainly involve three-dimensional modeling of the equipment in advance, virtual layout combined with site information, and then obtaining the final state of the equipment action by pulling the control instructions of the equipment. The three-dimensional simulation software processes the data inside the simulation software according to the starting state and the final state of the action, deduces the entire movement process, and then simulates and displays the movement process, so as to monitor according to the simulation display results. Since the simulation software itself is based on control instructions, the states during its operation are deduced by the simulation software, but the intermediate process may be inconsistent with the operation process of the real equipment, and the deduction process takes a certain amount of time, so it has a lag.

[0051] In order to solve the above technical problems, the embodiment of the present application provides a data processing method, which performs twin reconstruction on the operation status of the monitored device based on the acquired posture data of the monitored device at the current moment and the motion status of the joints included in the monitored device at the current moment. Since the posture data and motion status can reflect the real-time operation status of the monitored device, the operation status of the monitored device can be accurately and in real time restored, which is real-time. In addition, the twin reconstruction process restores the motion status of the monitored device in units of the joints and components included in the monitored device. For the monitored device including movable parts, the motion status of the monitored device can be restored more realistically and accurately, which is beneficial to the monitoring of the monitored device.

[0052] It should be noted that the data processing method provided in the embodiment of the present application can be applied to various scenarios using digital twin technology, such as equipment monitoring, digital twin, cloud technology, artificial intelligence, smart transportation, assisted driving, traffic simulation and other scenarios. The embodiment of the present application does not limit the application scenario of the data processing method. The embodiment of the present application will mainly introduce the equipment monitoring scenario as an example. The data processing method provided in the embodiment of the present application can restore the operating status of the monitored device in the monitored scene through the operating status of the instantiated object in the virtual scene, so as to perform visual monitoring of the monitored device based on the operating status of the instantiated object in the virtual scene, thereby realizing intelligent monitoring.

[0053] The data processing method provided in the embodiment of the present application can be executed by a computer device, which can be used as a device display terminal. The computer device can be, for example, a server or a terminal. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. Terminals include but are not limited to smart phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.

[0054] like Figure 1 As shown, Figure 1 An application scenario architecture diagram of a data processing method is shown. The application scenario is introduced by taking a computer device being a terminal as an example. The application scenario may include a terminal 100 .

[0055] The terminal 100 can be used as a device display terminal, also known as a visualization platform, for displaying virtual scenes, and by displaying the operation status of instantiated objects in the virtual scene, the operation status of the device to be monitored is reflected in real time, thereby monitoring the device to be monitored. In an embodiment of the present application, in order to display virtual scenes and perform multi-perspective, all-round stereoscopic monitoring, the visualization platform can be a three-dimensional (3D) visualization platform. A 3D visualization platform is a platform based on a three-dimensional rendering engine for displaying visual space scenes. The device display terminal or 3D visualization platform is pre-built, and uses a 3D game-like method through the 3D visualization platform to update and display the device status data processed by the data center in real time.

[0056] When it is necessary to perform visual monitoring on the monitored device, the operation status of the monitored device can be twin reconstructed, and the monitored device can be the device that needs to be monitored. Specifically, the terminal 100 can obtain the scene model data of the monitored scene, and obtain the device status data of the monitored device in the monitored scene, and the device status data includes the device identification of the monitored device, the posture data of the monitored device at the current moment, and the motion status of the joints included in the monitored device at the current moment. Among them, the monitored scene can be the area to which the monitored device belongs, and the scene model data can be obtained by collecting data for the monitored scene; the posture data can be data reflecting the posture status of the monitored device, and the motion status can reflect the motion status of the joints of the monitored device. The operation status of the monitored device can be reflected by the posture status and the motion status, so the posture data and the motion status can reflect the real-time operation status of the monitored device.

[0057] Then, the terminal 100 renders a virtual scene of the scene to be monitored according to the scene model data, and determines the instantiation object of the device to be monitored according to the device identifier of the device to be monitored. The rendered virtual scene can be seen in Figure 1As shown in 101. The instantiated object is generated by a first component description file, which is used to indicate the components included in the device to be monitored and the joints connecting the components. The joints connecting the components can reflect the connection relationship of the components included in the device to be monitored. The instantiated object is an object that can be displayed, and after rendering, it looks like the body of the device to be monitored. When the terminal is a 3D visualization platform, the virtual scene is a three-dimensional virtual scene.

[0058] In order to truly restore the operation status of the monitored device in the virtual scene, the terminal 100 can calculate the world transformation matrix corresponding to the monitored device based on the posture data of the monitored device. The main function of the world transformation matrix is ​​to convert the position and posture of the monitored device in the world coordinate system to the camera coordinate system or the screen coordinate system (that is, the coordinate system to which the virtual scene belongs) for rendering. Therefore, the terminal 100 can render the instantiated object in the virtual scene through the world transformation matrix, thereby restoring the posture of the monitored device in the monitored scene. For an example diagram of rendering the instantiated object in the virtual scene, please refer to Figure 1 As shown in 102.

[0059] Next, the terminal 100 can calculate the local transformation matrix of the joint according to the motion state of the joint included in the device to be monitored at the current moment. The main function of the local transformation matrix is ​​to convert the motion state of the joint in the world coordinate system to the camera coordinate system or the screen coordinate system for rendering. Therefore, the terminal 100 can determine the motion state of the virtual joint corresponding to the joint in the instantiated object at the current moment based on the local transformation matrix, and determine the motion state of the component model connected to the virtual joint in the instantiated object at the current moment based on the local transformation matrix, thereby restoring the operation of the device to be monitored in the monitored scene through the operation of the instantiated object in the virtual scene, so as to perform visual monitoring of the device to be monitored based on the operation of the instantiated object in the virtual scene. This realizes visual monitoring of equipment to help various enterprises to carry out digital transformation and improve production efficiency.

[0060] It should be noted that in the specific implementation of this application, user information and other related data may be involved in the entire process. When the above embodiments of this application are applied to specific products or technologies, the user's separate consent or separate permission is required, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0061] Next, the data processing method provided by the embodiment of the present application will be introduced from the perspective of the device display end in combination with the accompanying drawings. The device display end may be a terminal in a computer device, see Figure 2 , Figure 2 A flow chart of a data processing method is shown, the method comprising:

[0062] S201: Acquire scene model data of a scene to be monitored, and acquire device status data of a device to be monitored in the scene to be monitored.

[0063] The terminal can obtain the scene model data of the scene to be monitored, and obtain the device status data of the device to be monitored in the scene to be monitored. Among them, the scene to be monitored can be the area to which the device to be monitored belongs, and the scene model data can be the data obtained by collecting data for the scene to be monitored, which is used for twin reconstruction of the scene to be monitored. The device status data includes the device identification of the device to be monitored, the posture data of the device to be monitored at the current moment, and the motion state of the joints included in the device to be monitored at the current moment, which is used to twin reconstruct the device to be monitored so as to restore the operation status of the device to be monitored. Among them, the device identification can be the unique identification of the device to be monitored, and the current moment can be the moment when the motion state of the device to be monitored needs to be displayed.

[0064] It should be noted that the scene model data can be obtained based on the scene collection data collected in the data collection phase, and the device status data of the device to be monitored can be obtained based on the device real-time data obtained in the device real-time data access phase. In some cases, some sensitive information may be involved in the scene to be monitored (such as some factory production and operation management sites), and even some sensitive information may be involved in the device to be monitored. The sensitive information that may exist in the scene to be monitored can be referred to as the first target information, and the sensitive information that may be involved in the device real-time data of the device to be monitored can be referred to as the second target information. In this case, sensitive information may exist in the scene collection data collected in the data collection phase, and sensitive information may exist in the device real-time data obtained in the device real-time data access phase.

[0065] Based on this, in order to avoid the risk of leakage, sensitive information can be identified during the data collection stage. When it is determined that the first target information exists in the collected scene collection data, the first target information can be filtered out from the scene collection information to obtain the scene model data; when it is determined that the second target information exists in the acquired device real-time data, the second target information can be filtered out from the device real-time data to obtain the device status data. Among them, the method of identifying sensitive information can be manual review or automatic identification, which is not limited in the embodiments of the present application.

[0066] The embodiment of the present application filters the sensitive information in the scene acquisition data and the real-time data of the device, so that no sensitive information exists in the scene model data and the device status data acquired by the terminal, thereby avoiding the leakage of sensitive information.

[0067] It should be noted that there are many ways to collect scene acquisition data, including but not limited to oblique photography data, high-precision map tile data, automated building modeling data based on vector information, and manual refined modeling data. The above different data collection methods are usually applicable to different smart industry scenarios. For example, oblique photography data is suitable for data collection of indoor scenes to be monitored, such as workshops and factories; for another example, high-precision map tile data is suitable for data collection of outdoor scenes to be monitored, such as industrial docks and ports. Based on this, for different smart industry scenarios, when collecting data for the scene to be monitored to obtain scene acquisition data, the corresponding data collection method can be used for data collection.

[0068] In the real-time data access stage of the equipment, the real-time data of the equipment that needs to be accessed may include two categories, namely the posture data of the equipment to be monitored at the time of data collection and the motion data of the components included in the equipment to be monitored at the time of data collection. Among them, the posture data of the equipment to be monitored at the time of data collection includes the positioning data and attitude data of the equipment to be monitored at the time of data collection. The positioning data is obtained through the positioning device, and the attitude data is obtained through the gyroscope. Among them, the positioning device can be a global positioning system (Global Positioning System, GPS) installed on the equipment to be monitored, a camera deployed in the field, a positioning radar, etc. The positioning data can be represented in a variety of forms. For example, the positioning data can be represented in the form of longitude and latitude coordinates, or it can be represented by plane mapping in the area where the equipment to be monitored belongs, using two-dimensional plane coordinates. The attitude data can be represented in different ways, for example, it can be represented by Euler angles, rotation matrices, rotation vectors, etc.

[0069] The motion data of the components included in the monitored equipment at the time of data collection can be collected by sensors installed on the corresponding components, and then the motion data can be sent to the outside in real time. For example, it can be sent by using the fifth generation mobile communication technology (5th Generation Mobile Communication Technology, 5G), 4G and other network communications; or, the motion data of each component can be pulled by patrolling the status interface of the monitored equipment itself.

[0070] The above method obtains positioning data through positioning means such as GPS, and obtains motion data through sensors. Compared with the camera visual monitoring provided by related technologies, there is no need to deploy a large number of cameras in the area where the equipment is located, and the positioning problem can be solved by other means such as radar. Therefore, the leakage of sensitive information that may be caused by camera collection is avoided, and there is no problem of blind spots caused by cameras.

[0071] It can be understood that the terminal provided in the embodiment of the present application can be used as a 3D visualization platform. The data format that the terminal can support is subject to certain limitations. Therefore, in order to ensure that the terminal can process the device status data, the data format of the scene model data and device status data obtained by the terminal should be a data format that the terminal can support.

[0072] In a possible implementation, the device status data may be a device status structure that can be parsed by the terminal. The device status structure may be a serialized structure data using protocol buffers, which is simpler than JavaScript Object Notation (json) and eXtensible Markup Language (xml), and has a smaller amount of data after serialization and a faster serialization speed. The device status structure mainly includes the device identification, posture data, timestamp and device component status list of the device to be monitored, wherein the timestamp is used to indicate the time corresponding to the motion state, and the device component status list may include the motion state of the joints included in the device to be monitored at the current moment.

[0073] S202: Rendering a virtual scene of the scene to be monitored according to the scene model data, and determining an instantiated object of the device to be monitored according to the device identification of the device to be monitored.

[0074] After obtaining the scene model data, the terminal can render a virtual scene of the scene to be monitored based on the scene model data, thereby performing a twin reconstruction of the scene to be monitored on the terminal. In order to reconstruct the device to be monitored in the virtual scene, the device model of the device to be monitored can also be displayed in the virtual scene. The device model of the device to be monitored that is actually displayed in the virtual scene can be called an instantiated object. The instantiated object is generated by a first component description file, which is used to indicate the components included in the device to be monitored and the joints connecting the components, wherein the joints connecting the components are mainly used to reflect the connection relationship between the components.

[0075] It should be noted that the component description file is used to describe the adjacency relationship between the various components of the device, as well as a configuration file of the joints that the device can control. The format uses a json structure, which is easy to read and modify, and is used as important basic data in the subsequent process. The description file of the device to be monitored can be called the first component description file.

[0076] When the terminal is a 3D visualization platform, the virtual scene is a three-dimensional virtual scene, so that multi-viewing and all-round stereoscopic monitoring can be performed using the three-dimensional virtual scene as the visualization basis.

[0077] In an embodiment of the present application, there may be multiple ways to determine the instantiation object of the device to be monitored. One way may be to parse and assemble in real time based on the first component description file to obtain the instantiation object of the device to be monitored. Specifically, the terminal may obtain the first component description file of the device to be monitored according to the device identification of the device to be monitored, and then parse the first component description file to clarify the components included in the device to be monitored and the connection relationship between the components, and then obtain the component model of the components included in the device to be monitored and the virtual joints connecting the various component models, and the virtual joints reflect the connection relationship between the component models. Then, the terminal can assemble the component models of the components included in the device to be monitored according to the virtual joints connecting the various component models to obtain the device model of the device to be monitored, and determine the device model of the device to be monitored as the instantiation object of the device to be monitored. The instantiation object of the device to be monitored can be obtained by parsing and assembling in real time in the above manner.

[0078] Another way may be to store device templates of different devices in a template library in advance, and when it is necessary to obtain an instantiation object of a certain device, directly extract the device template from the template library to obtain the instantiation object. Specifically, the terminal can query the device template of the device to be monitored in the template library based on the device identification to obtain a template query result, and the template library includes device templates of devices indicated by different device identifications. Among them, the template query result is used to indicate whether there is a device template for the device to be monitored in the template library. In the two cases where there is a device template for the device to be monitored in the template library and there is no device template for the device to be monitored in the template library, the method of obtaining the instantiation object of the device to be monitored may be different, so the instantiation object of the device to be monitored can be further obtained based on the template query result. When the device status data is a device status structure, the device identification can be parsed from the device status structure.

[0079] If the template query result indicates that there is a device template for the device to be monitored in the template library, it means that the device to be monitored has been modeled in advance, the first component description file has been parsed, the device model of the device to be monitored has been assembled, and the device model is stored in the template library as a device template. Therefore, the terminal can determine the device template of the device to be monitored in the loaded template library as the instantiation object of the device to be monitored.

[0080] If the template query result indicates that there is no device template for the device to be monitored in the template library, it means that the device to be monitored has not been modeled in advance, and the device to be monitored may be newly introduced. In some cases, although the two devices are not the same device, since the device types of the two devices are the same, the appearance of the two devices may be exactly the same, and the two devices that look exactly the same can use the same device template. In this case, if the device status data also includes the device type of the device to be monitored, the device type of the device to be monitored can be further compared with the device type of the candidate device in the template library. If the comparison result indicates that the device type of the device to be monitored is consistent with the device type of the target device in the candidate device, it means that the device to be monitored and the target device look exactly the same, and the device templates of the two are the same, so the terminal can instantiate the device template of the target device to obtain the instantiation object of the device to be monitored. When the device status data is a device status structure, the device type can be parsed from the device status structure. When instantiating, the device identifier can be used as the name of the instantiation, and loaded in the terminal to obtain the instantiation object of the device to be monitored.

[0081] The above method determines the instantiation object of the device to be monitored by directly extracting the device template from the template library, thereby eliminating the need to repeatedly parse the first component description file and repeatedly assemble it, thereby improving the efficiency of determining the instantiation object and further improving the efficiency and real-time performance of visual monitoring.

[0082] It can be understood that the device template in the template library is obtained by modeling the device that may need to be monitored in advance. The device that may need to be monitored can be called a candidate device, and the description file of the candidate device can be called a second component description file. The terminal can obtain the second component description file of the candidate device, and the second component description file is obtained by integrating the information of the component model of the candidate device and the virtual joints connecting the component model. The component model of the candidate device and the virtual joints connecting the component model are obtained by decomposing and modeling the candidate device. Then, the second component description file is parsed to obtain the component model of the components included in the candidate device and the virtual joints connecting the component model, and then the component model of the components included in the candidate device is assembled according to the virtual joints connecting the component model to obtain the device model of the candidate device. After the device model is rendered by the visualization platform, it should look like the candidate device body. The assembled device model can not be directly loaded and rendered on the visualization platform, but the device model of the candidate device can be used as a device template, and the device template of the candidate device and the device identification of the candidate device are stored in the template library in correspondence, so that it can be directly extracted and used later.

[0083] In the above process, the candidate devices are decomposed and modeled. The specific implementation method of this step is that when modeling and restoring the candidate devices that may need to be monitored, a single complete model output is not used, but the entire model is disassembled according to the number of movable parts in the daily use of the candidate devices. The component models of the various components of the candidate device are obtained by disassembling them into individual components and modeling them separately. The connections between the disassembled components are used as joints. For the component model, the connections between the component models can be defined as virtual joints, which must comply with the principle of forward kinematics so that the component model can correctly simulate the movement of the component in the candidate device by performing matrix transformation on the virtual joints.

[0084] Based on the component models and virtual joints obtained by decomposition modeling, the second component description file of the candidate device can be output. Specifically, based on the component models and virtual joints obtained by decomposition modeling, information can be integrated in a specific data structure, and the connection points of the movable components can be defined as joints and named, so as to obtain the second component description file. Take a simple robotic arm structure as an example, such as Figure 3 As shown, through the above process, the robot arm structure can be split into Figure 3 The 8 disassembled parts are the bottom trolley, the rotatable base, the rotating shaft 1, the arm 1, the rotating shaft 2, the arm 2, the rotating shaft 3 and the manipulator.

[0085] According to the analysis, its internal movement can be obtained through Figure 4 The four joint controls shown are joint 1, joint 2, joint 3, and joint 4. The joints can reflect the connection relationship between the components and are used to control the movement of the components. Among them, joint 1 is used to control the horizontal rotation of the rotatable base and its connected components, joint 2 is used to control the rotation of the arm and its connected components, joint 3 is used to control the rotation of the arm and its connected components, and joint 4 is used to control the rotation of the manipulator.

[0086] Then the second component description file of this candidate device can be represented by a tree structure in the json structure, thereby reflecting the various components included in the candidate device and the connection relationship between the components, see Figure 5 As shown. Figure 5 , starting from the root joint, from Figure 3 The bottom to top of the robot arm structure shown are connected in sequence to joint 1, joint 2, joint 3 and joint 4. Among them, joint 1 is connected to the rotatable base, joint 2 is connected to the rotation axis 1 and arm 1, joint 3 is connected to the rotation axis 2 and arm 2, and joint 4 is connected to the rotation axis 3 and the manipulator. In addition, the root joint is also connected to 4 wheel joints and the bottom trolley. Each wheel joint is connected to the corresponding wheel, which is mainly used to control the position and posture of the bottom trolley.

[0087] The embodiment of the present application stores the device model as a device template in a template library. When the instantiation object of the device to be monitored needs to be determined later, it can be directly extracted from the template library without repeated parsing of the first component description file and repeated assembly, thereby improving the efficiency of determining the instantiation object and further improving the efficiency and real-time performance of visual monitoring.

[0088] It is understandable that the candidate devices may include the device to be monitored, or may not include the device to be monitored. When the device identification of the device to be monitored is queried in the template library, the template query result may indicate that the device template of the device to be monitored exists in the template library, or may indicate that the device template of the device to be monitored does not exist in the template library.

[0089] It should be noted that the decomposition modeling of the candidate device and the generation of the second component description file can be performed in the data collection stage.

[0090] S203: Calculate a world transformation matrix corresponding to the device to be monitored according to the posture data of the device to be monitored, and render the instantiated object in the virtual scene using the world transformation matrix.

[0091] After obtaining the instantiated object of the device to be monitored, the next step is to display the instantiated object in the virtual scene. How the instantiated object is displayed in the virtual scene mainly depends on the posture data and motion state of the device to be monitored. The posture data can determine the posture of the instantiated object of the device to be monitored in the virtual scene, and the motion state can determine the motion of the virtual joints and component models of the device to be monitored in the virtual scene. Therefore, the terminal can first calculate the world transformation matrix corresponding to the device to be monitored based on the posture data of the device to be monitored. The main function of the world transformation matrix is ​​to convert the position and posture of the device to be monitored in the world coordinate system to the camera coordinate system or the screen coordinate system (that is, the coordinate system to which the virtual scene belongs) for rendering. Therefore, the terminal can render the instantiated object in the virtual scene through the world transformation matrix, and place the instantiated object in the correct position in the virtual scene, thereby restoring the posture of the device to be monitored in the scene to be monitored.

[0092] S204. Calculate the local transformation matrix of the joint according to the motion state of the joint included in the device to be monitored at the current moment, and determine the motion state of the virtual joint corresponding to the joint in the instantiated object at the current moment based on the local transformation matrix, and determine the motion state of the component model connected to the virtual joint in the instantiated object at the current moment based on the local transformation matrix.

[0093] Next, the terminal can calculate the local transformation matrix of the joints according to the motion state of the joints included in the device to be monitored at the current moment. The main function of the local transformation matrix is ​​to convert the motion state of the joints in the world coordinate system to the camera coordinate system or the screen coordinate system for rendering. Therefore, the terminal can determine the motion state of the virtual joints corresponding to the joints in the instantiated object at the current moment based on the local transformation matrix, and determine the motion state of the component model connected to the virtual joints in the instantiated object at the current moment based on the local transformation matrix, thereby restoring the operation of the device to be monitored in the monitored scene through the operation of the instantiated object in the virtual scene, so as to perform visual monitoring of the device to be monitored based on the operation of the instantiated object in the virtual scene.

[0094] When the device status data is a device status structure, the device status structure includes a device component status list, and the device component status list records the name and status of each joint of the device to be monitored. Therefore, the terminal can obtain the name and status of each joint in the device to be monitored according to the device component status list in the device status structure. Since the instantiation object of the device to be monitored is generated by the first component description file, the corresponding virtual joint of the instantiation object of the device to be monitored can be directly found by the joint name. Through the real-time motion state of the joint, such as the angular rotation along a certain axis and the displacement along a certain axis, a local transformation matrix can be calculated and applied to the virtual joint. Correspondingly, the component model connected by the virtual joint is also affected by the local transformation matrix, thereby obtaining the motion state of the connected component model. The certain axis here refers to one of the three axes X, Y, and Z of the common three-dimensional coordinate system.

[0095] It can be seen from the above technical solution that when it is necessary to perform visual monitoring on the monitored device, the operation of the monitored device can be twinned and reconstructed. Specifically, the scene model data of the monitored scene can be obtained, and the device status data of the monitored device in the monitored scene can be obtained. The device status data includes the device identification of the monitored device, the posture data of the monitored device at the current moment, and the motion state of the joints included in the monitored device at the current moment. The posture data and the motion state can reflect the real-time operation of the monitored device. Then, the virtual scene of the monitored scene is rendered according to the scene model data, and the instantiated object of the monitored device is determined according to the device identification of the monitored device. The instantiated object is generated by a first component description file, and the first component description file is used to indicate the components included in the monitored device and the joints connecting the components. The instantiated object is an object that can be displayed, and after rendering, it looks like the body of the monitored device. In order to truly restore the operation of the monitored device in the monitored scene in the virtual scene, the world transformation matrix corresponding to the monitored device can be calculated according to the posture data of the monitored device, and the instantiated object is rendered in the virtual scene through the world transformation matrix, thereby restoring the posture of the monitored device in the monitored scene. Then, according to the motion state of the joints included in the device to be monitored at the current moment, the local transformation matrix of the joints is calculated, and the motion state of the virtual joints corresponding to the joints in the instantiated object at the current moment is determined based on the local transformation matrix, and the motion state of the component model connected to the virtual joints in the instantiated object at the current moment is determined based on the local transformation matrix, so as to restore the operation of the device to be monitored in the monitored scene through the operation of the instantiated object in the virtual scene, so as to perform visual monitoring of the device to be monitored based on the operation of the instantiated object in the virtual scene. Based on the acquired posture data of the device to be monitored at the current moment and the motion state of the joints included in the device to be monitored at the current moment, the operation of the device to be monitored is twin-reconstructed. Since the posture data and motion state can reflect the real-time operation of the device to be monitored, that is, the data used to control the instantiated object comes from the real-time operation of the device to be monitored, no deduction is required, so the operation of the device to be monitored can be accurately and in real time restored, which is real-time. Moreover, the twin reconstruction process restores the motion state of the device to be monitored based on the joints and components included in the device to be monitored. For the device to be monitored that includes movable parts, the motion state of the device to be monitored can be restored more realistically and accurately, which is conducive to the monitoring of the device to be monitored.

[0096] In the embodiments of the present application, the main purpose is to reconstruct the operation status of the monitored equipment by twinning, so as to display the operation status of the monitored equipment in the actual monitored scene in the virtual scene, thereby realizing the visual monitoring of the equipment based on digital twins. The equipment status data of the monitored equipment is the key factor for accurate twin reconstruction. The following will introduce the generation method of equipment status data.

[0097] In the embodiment of the present application, the device status data is derived from the collected real-time data of the device. Under normal circumstances, the real-time data of the device to be monitored can be collected once every data collection moment. If the collection frequency is very high, the interval between two data collection moments is very short, and the current moment showing the motion state of the device to be monitored can be the data collection moment. At this time, the device status data of the device to be monitored can be the initial state data, and the initial state data is obtained by format conversion of the real-time data of the device to be monitored collected by the device to be monitored at the data collection moment based on the first component description file of the device to be monitored. That is, the initial state data is data that meets the format requirements of the visualization platform obtained by format conversion based on the real-time data of the device. When the device status data is a device status structure, the initial state data is also a device status structure.

[0098] In some cases, the acquisition frequency may be slightly low, and there is a certain time interval between two data acquisition moments. In this case, no data is collected at the moment between the two data acquisition moments. If only the real-time data of the device collected at the two data acquisition moments are used for twin reconstruction, the motion state may suddenly jump. Based on this, if the current moment showing the motion state of the device to be monitored is the moment between the two data acquisition moments, the device state data of the device to be monitored can be predicted based on the initial state data, or the device state data of the device to be monitored is calculated by the device data model of the simulation system on the initial state data.

[0099] The prediction here can refer to the dynamic prediction technology used in the gaming industry. After obtaining the initial state data, since the initial state data is only obtained by format conversion of the device real-time data, the device real-time data is obtained by data collection at the data collection time. In this way, in order to obtain the device state data at the current moment, the initial state data can be dynamically predicted to obtain the device state data, thereby filling in the device state data corresponding to the moment between the two data collection moments. In this way, when rendering, not only can the motion state of the device to be monitored corresponding to the data collection moment be displayed, but also the motion state of the device to be monitored corresponding to the interval between the two data collection moments can be displayed, thereby avoiding sudden jumps in the motion state of the device to be monitored and achieving a smooth transition of the motion state.

[0100] When the device status data is a device status structure, dynamically predicting the initial status data to obtain the device status data is actually predicting various parameters in the device status structure based on the initial status data, such as device identification, posture data, timestamp, and device component status list.

[0101] Correspondingly, for the key frame of "initial state data" at the time of data collection, in order to avoid the delay caused by related processing in the process of displaying the motion state of the monitored equipment, the initial state data at the time of data collection can be corrected and delayed to compensate, so as to obtain more accurate equipment state data, so that the motion state of the monitored equipment at each moment can be more accurately restored.

[0102] It should be noted that in the scenario of prediction and delay compensation, the obtained device state data is processed state data relative to the initial state data. The device state data is completely identical to the initial state data in structure, and the content is the device state structure. However, as the actual acquired data, the device state data is acquired at a higher frequency. For example, it can be acquired at a frequency of more than 25 frames per second to display its corresponding motion state.

[0103] It should be noted that the device status data can be generated in the data processing stage, and the data processing stage can be completed in the data middle station. The data middle station and the terminal (device display terminal or visualization platform) can be independent of each other, thereby separating data processing from visualization display. In this case, the way for the terminal to obtain the device status data of the device to be monitored can be that the terminal subscribes to the device status data generated by the data middle station, and the data middle station distributes the device status data to the terminal. The data middle station is pre-deployed and constructed, and is responsible for collecting the real-time data of each device to be monitored, processing the real-time data of the device, and then publishing it to the outside in the form of real-time broadcast.

[0104] In the embodiment of the present application, data processing is separated from visual display. The device display terminal does not need to calculate the device status data, but directly obtains the device status data for rendering, thereby reducing the amount of computation on the terminal, so the hardware cost is relatively lower.

[0105] When data processing is separated from visual display, the data center load accesses the real-time data of the device to be monitored, and generates device status data based on the real-time data of the device, thereby distributing the device status data to the terminal. Among them, the device to be monitored can be one or more, and the embodiment of the present application does not limit this.

[0106] Next, the data processing method provided by the embodiment of the present application is introduced from the perspective of the data middle station. The data middle station can be a server or a terminal different from the terminal used by the device display end.

[0107] See also Figure 6 As shown, the method includes:

[0108] S601: Acquire real-time data of a device to be monitored, and acquire a first component description file of the device to be monitored.

[0109] In the real-time data access stage of the device, the real-time data of the device can be accessed to the data middle station. The real-time data of the device that needs to be accessed in this process can include two categories, namely the posture data of the device to be monitored at the time of data collection and the motion data of the components included in the device to be monitored at the time of data collection. Among them, the posture data of the device to be monitored at the time of data collection includes the positioning data and attitude data of the device to be monitored at the time of data collection. The positioning data is obtained by the positioning device, and the attitude data is obtained by the gyroscope. Among them, the positioning device can be a GPS installed on the device to be monitored, a camera deployed in the field, a positioning radar, etc. The positioning data can be expressed in a variety of forms. For example, the positioning data can be expressed in the form of longitude and latitude coordinates, or it can be mapped in the area where the device to be monitored belongs, and expressed using two-dimensional plane coordinates. The attitude data can be expressed in different ways, for example, it can be expressed by Euler angles, rotation matrices, rotation vectors, etc.

[0110] The motion data of the components included in the monitored equipment at the time of data collection can be collected by sensors installed on the corresponding components, and then the motion data can be sent to the outside in real time. For example, it can be sent by using 5G, 4G and other network communications; or, the motion data of each component can be pulled by patrolling the status interface of the monitored equipment itself.

[0111] The first component description file can be generated during the data collection stage. When it is determined that the monitored device needs to be monitored, the monitored device can be decomposed and modeled. The specific implementation method of this step is that when modeling and restoring the monitored device, a single complete model output method is not adopted, but the entire model is disassembled according to the number of movable parts in the daily use process of the monitored device. The component models of the various components of the monitored device are obtained by disassembling them into individual components and modeling them separately. The connection points of the disassembled components are used as joints. For the component model, the connection points of the component model can be defined as virtual joints, which must comply with the principle of forward kinematics so that the component model can correctly simulate the movement of the component in the monitored device by performing matrix transformation on the virtual joints.

[0112] The first component description file of the device to be monitored can be output based on the component models and virtual joints obtained by decomposition modeling. Specifically, the information can be integrated in a specific data structure based on the component models and virtual joints obtained by decomposition modeling, and the connection points of the movable components can be defined as joints and named, thereby obtaining the first component description file. The specific implementation method of decomposition modeling and generating the first description file is similar to the specific implementation method of generating the second description file, which will not be repeated here.

[0113] S602: Convert the format of the real-time data of the device based on the first component description file to obtain initial state data.

[0114] The data center converts the first component description file into initial state data that can be parsed by the terminal (3D visualization platform) in combination with the real-time data of the device and a timestamp (for data synchronization).

[0115] S603: Determine device status data of the device to be monitored according to the initial status data.

[0116] Next, the data center determines the device status data of the device to be monitored based on the initial status data, wherein the device status data includes the device identification of the device to be monitored, the posture data of the device to be monitored at the current moment, and the motion state of the joints included in the device to be monitored at the current moment.

[0117] In a possible implementation, the real-time data of the device is collected at the data collection time, and the method of determining the device status data of the device to be monitored based on the initial status data can be that if the current moment is the data collection moment, the initial status data is determined as the device status data; if the current moment is between two data collection moments, the device status data is obtained by prediction based on the initial status data, or the device status data is obtained by calculating the initial status data through the device data model of the simulation system. The specific implementation of this process can be referred to the above introduction, and will not be repeated here.

[0118] S604: Send the device status data of the device to be monitored to the device display terminal.

[0119] After generating the device status data, the data middle platform can distribute the device status data to the device display end so that the device display end can reconstruct the twin of the monitored device based on the device status data and realize device visualization monitoring based on data-driven digital twins.

[0120] The device status data used for twin reconstruction of the monitored device in the embodiment of the present application is based on the real-time data from the device, so that it can reflect the real-time operation of the monitored device, that is, the data used to control the instantiated object is derived from the real-time operation of the monitored device, without the need for deduction, so the operation of the monitored device can be accurately and in real time restored, with real-time performance. In addition, the twin reconstruction process restores the motion state of the monitored device based on the joints and components included in the monitored device. For the monitored device including movable components, the motion state of the monitored device can be restored more realistically and accurately, which is beneficial to the monitoring of the monitored device.

[0121] Through the above introduction, in order to realize visual monitoring of equipment, the data processing method provided in the embodiment of the present application mainly includes four stages: Figure 7 As shown, they are the data collection stage, the equipment real-time data access stage, the data processing stage and the visualization stage.

[0122] In the data collection phase, it mainly includes scene 3D reconstruction (see Figure 7 S701), equipment decomposition modeling (see Figure 7 S702), generate a second device description file (see Figure 7 In S703). In S701, data is collected for the area to which the candidate device belongs, and scene collection data is output. After the collection is completed, the scene collection data is converted into a model format supported by the 3D visualization platform to obtain scene model data. In S702, the entire model is disassembled according to the number of movable parts in the daily use of the candidate device. Disassemble it into individual components and model them separately to obtain component models of each component of the candidate device. The connection points of the disassembled components are used as joints. For the component model, the connection points of the component model can be defined as virtual joints, which must comply with the principle of forward kinematics so that the component model can correctly simulate the movement of the component in the candidate device by performing matrix transformation on the virtual joints. In S703, based on the component models and virtual joints obtained by decomposition modeling, information is integrated in a specific data structure, and the connection points of the movable parts are defined as joints and named, thereby obtaining a second component description file.

[0123] In the real-time data access stage, the main work is real-time data access (see Figure 7 In S704). In the device real-time data access stage, the device real-time data of at least one device to be monitored can be accessed to the data middle platform, and the device real-time data accessed to the data middle platform is discrete single device real-time data.

[0124] In the data processing stage, including data integration processing (see Figure 7 S705) and data distribution (see Figure 7 S706 in the data integration process). The data integration process is mainly implemented by the data middle station integrating the posture data of the device to be monitored and the motion data of the components. If the candidate device includes the device to be monitored, based on the device identification of the device to be monitored, the second component description file of the candidate device is searched, and the second component description file found is used as the first component description file of the device to be monitored; if the candidate device does not include the device to be monitored, when it is determined that the device to be monitored needs to be monitored, data collection can also be performed on the device to be monitored, so as to generate the first component description file of the device to be monitored in the data collection stage of the device to be monitored, and send the first component description file to the data middle station. Among them, the generation method of the first component description file is similar to the generation method of the second component description file, which will not be repeated here. Based on the first component description file of the device to be monitored, the data middle station converts the real-time data of the device to be monitored into the initial state data. The initial state data is a device state structure that can be parsed by the 3D visualization platform. By analyzing the initial state data of multiple frames, the dynamic prediction technology and delay compensation technology used in the game industry are used to predict the various parameters in the device state structure when receiving the initial state data interval, and correct and compensate for the key frame of the received initial state data. In this way, a series of processed initial state data can be obtained, that is, the device state data of the device to be monitored. Then, the device state data is distributed to the outside at a frequency of more than 25 frames per second, and the device state data is subscribed and consumed by the 3D visualization platform.

[0125] In the visualization stage, the main task is to reconstruct the twin of the monitored device in the virtual scene, so as to display the instantiated object of the monitored device in real time (see Figure 7 S707 in the figure). The specific implementation method of this process is mainly to use the scene model data to render the virtual scene of the scene to be monitored. The first component description file is parsed to clarify the components included in the device to be monitored and the connection relationship between the components, so as to obtain the component models of the components included in the device to be monitored and the virtual joints connecting the various component models. According to the virtual joints connecting the various component models, the component models of the components included in the device to be monitored are assembled to obtain the device model of the device to be monitored. The device model of the device to be monitored is used as a device template, and the device template of the device to be monitored and the device identification of the device to be monitored are stored in the template library in correspondence so that they can be directly extracted and used later.

[0126] Subscribe to the device status data obtained in the data processing phase, and determine whether the 3D visualization platform has a device template for the device to be monitored based on the device identifier included in the device status data. If so, obtain the instantiated object loaded in the 3D visualization platform to obtain the instantiated object of the device to be monitored. If not, further determine the instantiated object of the device to be monitored based on the device type of the device to be monitored.

[0127] According to the posture data in the device status data, the instantiated object is placed in the correct position in the virtual scene. Then, according to the motion state of the joints included in the monitored device at the current moment, the local transformation matrix of the joints is calculated, and the motion state of the virtual joints corresponding to the joints in the instantiated object at the current moment is determined based on the local transformation matrix, and the motion state of the component model connected to the virtual joints in the instantiated object at the current moment is determined based on the local transformation matrix.

[0128] After the above process, device visualization monitoring based on data-driven digital twins can be realized for the equipment to be monitored in the 3D visualization platform.

[0129] It should be noted that, based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0130] based on Figure 2 Corresponding to the data processing method provided in the embodiment, the embodiment of the present application also provides a data processing device 800. Figure 8 As shown, the data processing device 800 includes an acquisition unit 801, a rendering unit 802, a determination unit 803 and a calculation unit 804:

[0131] The acquisition unit 801 is used to acquire scene model data of a scene to be monitored, and to acquire device status data of a device to be monitored in the scene to be monitored, wherein the device status data includes a device identifier of the device to be monitored, position data of the device to be monitored at a current moment, and a motion state of a joint included in the device to be monitored at the current moment;

[0132] The rendering unit 802 is used to render a virtual scene of the scene to be monitored according to the scene model data;

[0133] The determining unit 803 is used to determine the instantiation object of the device to be monitored according to the device identification of the device to be monitored, wherein the instantiation object is generated by a first component description file, and the first component description file is used to indicate the components included in the device to be monitored and the joints connecting the components;

[0134] The calculation unit 804 is used to calculate the world transformation matrix corresponding to the device to be monitored according to the posture data of the device to be monitored;

[0135] The rendering unit 802 is further configured to render the instantiated object in the virtual scene using the world transformation matrix;

[0136] The calculation unit 804 is further used to calculate the local transformation matrix of the joint according to the motion state of the joint included in the device to be monitored at the current moment;

[0137] The determination unit 803 is also used to determine the motion state of the virtual joint corresponding to the joint in the instantiated object at the current moment based on the local transformation matrix, and to determine the motion state of the component model connected to the virtual joint in the instantiated object at the current moment based on the local transformation matrix.

[0138] In a possible implementation manner, the determining unit 803 is configured to:

[0139] Acquire a first component description file of the device to be monitored according to the device identification of the device to be monitored;

[0140] Parsing the first component description file to obtain component models of components included in the device to be monitored and virtual joints connecting the component models;

[0141] According to the virtual joints connecting the component models, the component models of the components included in the device to be monitored are assembled to obtain the device model of the device to be monitored, and the device model of the device to be monitored is determined as the instantiation object of the device to be monitored.

[0142] In a possible implementation manner, the determining unit 803 is configured to:

[0143] Based on the device identification, querying the device template of the device to be monitored in the template library to obtain a template query result, wherein the template library includes device templates of candidate devices indicated by different device identifications;

[0144] An instantiated object of the device to be monitored is obtained based on the template query result.

[0145] In a possible implementation manner, the determining unit 803 is configured to:

[0146] If the template query result indicates that the device template of the device to be monitored exists in the template library, the device template of the device to be monitored in the loaded template library is determined as the instantiation object of the device to be monitored.

[0147] In a possible implementation, the device status data further includes the device type of the device to be monitored, and the determining unit 803 is configured to:

[0148] If the template query result indicates that the device template of the device to be monitored does not exist in the template library, then comparing the device type of the device to be monitored with the device types of candidate devices in the template library;

[0149] If the comparison result indicates that the device type of the device to be monitored is consistent with the device type of the target device among the candidate devices, the device template of the target device is instantiated to obtain the instantiated object of the device to be monitored.

[0150] In a possible implementation, the device further includes a parsing unit, an assembling unit, and a storage unit:

[0151] The acquisition unit 801 is further used to acquire a second component description file of the candidate device, wherein the second component description file is obtained by integrating information of a component model of the candidate device and a virtual joint connecting the component models, wherein the component model of the candidate device and the virtual joint connecting the component models are obtained by decomposing and modeling the candidate device;

[0152] The parsing unit is used to parse the second component description file to obtain component models of the components included in the candidate device and virtual joints connecting the component models;

[0153] The assembling unit is used to assemble the component models of the components included in the candidate device according to the virtual joints connecting the component models to obtain the device model of the candidate device;

[0154] The storage unit is used to use the device model of the candidate device as a device template, and store the device template of the candidate device and the device identification of the candidate device in the template library in correspondence.

[0155] In a possible implementation, if the current moment is a data collection moment, the device state data of the device to be monitored is initial state data, and the initial state data is obtained by format conversion of device real-time data collected by the device to be monitored at the data collection moment based on a first component description file of the device to be monitored;

[0156] If the current moment is between two data collection moments, the device status data of the device to be monitored is predicted based on the initial status data, or the device status data of the device to be monitored is calculated on the initial status data through the device data model of the simulation system.

[0157] In a possible implementation, the scene model data is obtained by filtering out the first target information from the scene acquisition data, and the device status data of the device to be monitored is obtained by filtering out the second target information from the device real-time data.

[0158] It can be seen from the above technical solution that when it is necessary to perform visual monitoring on the monitored device, the operation of the monitored device can be twinned and reconstructed. Specifically, the scene model data of the monitored scene can be obtained, and the device status data of the monitored device in the monitored scene can be obtained. The device status data includes the device identification of the monitored device, the posture data of the monitored device at the current moment, and the motion state of the joints included in the monitored device at the current moment. The posture data and the motion state can reflect the real-time operation of the monitored device. Then, the virtual scene of the monitored scene is rendered according to the scene model data, and the instantiated object of the monitored device is determined according to the device identification of the monitored device. The instantiated object is generated by a first component description file, and the first component description file is used to indicate the components included in the monitored device and the joints connecting the components. The instantiated object is an object that can be displayed, and after rendering, it looks like the body of the monitored device. In order to truly restore the operation of the monitored device in the monitored scene in the virtual scene, the world transformation matrix corresponding to the monitored device can be calculated according to the posture data of the monitored device, and the instantiated object is rendered in the virtual scene through the world transformation matrix, thereby restoring the posture of the monitored device in the monitored scene. Then, according to the motion state of the joints included in the device to be monitored at the current moment, the local transformation matrix of the joints is calculated, and the motion state of the virtual joints corresponding to the joints in the instantiated object at the current moment is determined based on the local transformation matrix, and the motion state of the component model connected to the virtual joints in the instantiated object at the current moment is determined based on the local transformation matrix, so as to restore the operation of the device to be monitored in the monitored scene through the operation of the instantiated object in the virtual scene, so as to perform visual monitoring of the device to be monitored based on the operation of the instantiated object in the virtual scene. Based on the acquired posture data of the device to be monitored at the current moment and the motion state of the joints included in the device to be monitored at the current moment, the operation of the device to be monitored is twin-reconstructed. Since the posture data and motion state can reflect the real-time operation of the device to be monitored, that is, the data used to control the instantiated object comes from the real-time operation of the device to be monitored, no deduction is required, so the operation of the device to be monitored can be accurately and in real time restored, which is real-time. Moreover, the twin reconstruction process restores the motion state of the device to be monitored based on the joints and components included in the device to be monitored. For the device to be monitored that includes movable parts, the motion state of the device to be monitored can be restored more realistically and accurately, which is conducive to the monitoring of the device to be monitored.

[0159] based on Figure 6Corresponding to the data processing method provided in the embodiment, the embodiment of the present application also provides a data processing device 900. Fig. 9 As shown, the data processing device 900 includes an acquisition unit 901, a conversion unit 902, a determination unit 903 and a sending unit 904:

[0160] The acquisition unit 901 is used to acquire the real-time data of the device to be monitored collected at the data collection time, and to acquire the first component description file of the device to be monitored;

[0161] The conversion unit 902 is used to convert the format of the device real-time data based on the first component description file to obtain initial state data;

[0162] The determining unit 903 is used to determine the device state data of the device to be monitored according to the initial state data, wherein the device state data includes the device identification of the device to be monitored, the position data of the device to be monitored at the current moment, and the motion state of the joints included in the device to be monitored at the current moment;

[0163] The sending unit 904 is used to send the device status data of the device to be monitored to the device display terminal.

[0164] In a possible implementation, the device real-time data is collected at a data collection moment, and the determining unit 903 is used to determine the initial state data as the device state data if the current moment is the data collection moment;

[0165] If the current moment is a moment between two data collection moments, the device state data is obtained by predicting based on the initial state data, or by calculating the initial state data through a device data model of a simulation system.

[0166] In a possible implementation, the real-time device data includes the position data of the device to be monitored at the data collection time and the motion data of the components included in the device to be monitored at the data collection time;

[0167] The position and posture data of the device to be monitored at the data collection time includes positioning data and posture data of the device to be monitored at the data collection time, the positioning data is obtained by a positioning device, and the posture data is obtained by a gyroscope;

[0168] The motion data of the components included in the equipment to be monitored at the time of data collection are collected by sensors installed on the corresponding components.

[0169] The embodiment of the present application also provides a computer device, which can execute the data processing method. The computer device can be a terminal, taking a smart phone as an example:

[0170] Fig.10 The block diagram shows a partial structure of a smart phone provided in an embodiment of the present application. Fig.10 , the smartphone includes: a radio frequency (full name: Radio Frequency, English abbreviation: RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (English abbreviation: WiFi) module 1070, a processor 1080, and a power supply 1090. The input unit 1030 may include a touch panel 1031 and other input devices 1032, the display unit 1040 may include a display panel 1041, and the audio circuit 1060 may include a speaker 1061 and a microphone 1062. It can be understood that, Fig.10 The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0171] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0172] The processor 1080 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0173] In this embodiment, the processor 1080 in the smart phone can execute the data processing method provided in each embodiment of the present application.

[0174] The computer device provided in the embodiment of the present application may also be a server. Fig.11 As shown, Fig.11 The structural diagram of the server 1100 provided in the embodiment of the present application, the server 1100 may have relatively large differences due to different configurations or performances, and may include one or more processors, such as a central processing unit (CPU) 1122, and a memory 1132, one or more storage media 1130 (such as one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1122 can be configured to communicate with the storage medium 1130 and execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0175] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0176] In this embodiment, the steps to be performed by the CPU 1122 in the server 1100 may be based on Fig.11 The structure shown is implemented.

[0177] According to one aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the data processing method described in the above-mentioned embodiments.

[0178] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional implementations of the above-mentioned embodiments.

[0179] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0180] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a terminal, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store computer programs.

[0185] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technical members in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire scene model data of a scene to be monitored, and acquire device status data of a device to be monitored in the scene to be monitored, wherein the device status data includes a device identification of the device to be monitored, position data of the device to be monitored at a current moment, and a motion state of a joint included in the device to be monitored at the current moment; Rendering a virtual scene of the scene to be monitored according to the scene model data, and determining an instantiated object of the device to be monitored according to the device identification of the device to be monitored, wherein the instantiated object is generated by a first component description file, and the first component description file is used to indicate the components included in the device to be monitored and the joints connecting the components; Calculate the world transformation matrix corresponding to the device to be monitored according to the posture data of the device to be monitored, and render the instantiated object in the virtual scene through the world transformation matrix; According to the motion state of the joint included in the device to be monitored at the current moment, the local transformation matrix of the joint is calculated, and the motion state of the virtual joint corresponding to the joint in the instantiated object at the current moment is determined based on the local transformation matrix, and the motion state of the component model connected to the virtual joint in the instantiated object at the current moment is determined based on the local transformation matrix.

2. The method according to claim 1, characterized in that: The step of determining the instantiation object of the device to be monitored according to the device identification of the device to be monitored includes: Acquire a first component description file of the device to be monitored according to the device identification of the device to be monitored; Parsing the first component description file to obtain component models of components included in the device to be monitored and virtual joints connecting the component models; According to the virtual joints connecting the component models, the component models of the components included in the device to be monitored are assembled to obtain the device model of the device to be monitored, and the device model of the device to be monitored is determined as the instantiation object of the device to be monitored.

3. The method according to claim 1, characterized in that The step of determining the instantiation object of the device to be monitored according to the device identification of the device to be monitored includes: Based on the device identification, querying the device template of the device to be monitored in the template library to obtain a template query result, wherein the template library includes device templates of candidate devices indicated by different device identifications; An instantiated object of the device to be monitored is obtained based on the template query result.

4. The method according to claim 3, characterized in that The obtaining the instantiated object of the device to be monitored based on the template query result includes: If the template query result indicates that the device template of the device to be monitored exists in the template library, the device template of the device to be monitored in the template library loaded is determined as the instantiation object of the device to be monitored.

5. The method according to claim 4, characterized in that The device status data also includes the device type of the device to be monitored, and the method further includes: If the template query result indicates that the device template of the device to be monitored does not exist in the template library, then comparing the device type of the device to be monitored with the device types of candidate devices in the template library; If the comparison result indicates that the device type of the device to be monitored is consistent with the device type of the target device among the candidate devices, the device template of the target device is instantiated to obtain the instantiated object of the device to be monitored.

6. The method according to claim 3, characterized in that The method further comprises: Acquire a second component description file of the candidate device, where the second component description file is obtained by integrating information of a component model of the candidate device and a virtual joint connecting the component models, where the component model of the candidate device and the virtual joint connecting the component models are obtained by decomposing and modeling the candidate device; Parsing the second component description file to obtain component models of components included in the candidate device and virtual joints connecting the component models; According to the virtual joints connecting the component models, the component models of the components included in the candidate device are assembled to obtain the device model of the candidate device, and the device model of the candidate device is used as a device template. The device template of the candidate device and the device identification of the candidate device are stored in the template library in correspondence.

7. The method according to claim 1, characterized in that If the current moment is a data collection moment, the device state data of the device to be monitored is initial state data, and the initial state data is obtained by format conversion of the device real-time data collected by the device to be monitored at the data collection moment based on the first component description file of the device to be monitored; If the current moment is between two data collection moments, the device status data of the device to be monitored is predicted based on the initial status data, or the device status data of the device to be monitored is calculated on the initial status data through the device data model of the simulation system.

8. The method according to claim 1, characterized in that The scene model data is obtained by filtering out the first target information from the scene acquisition data, and the device status data of the device to be monitored is obtained by filtering out the second target information from the device real-time data.

9. A data processing method, characterized in that: The method comprises: Acquire real-time data of a device to be monitored, and acquire a first component description file of the device to be monitored; Performing format conversion on the device real-time data based on the first component description file to obtain initial state data; Determine device status data of the device to be monitored according to the initial status data, wherein the device status data includes a device identification of the device to be monitored, position data of the device to be monitored at a current moment, and a motion state of a joint included in the device to be monitored at the current moment; Send the device status data of the device to be monitored to the device display terminal.

10. The method according to claim 9, characterized in that The real-time data of the device is collected at the data collection time, and the device status data of the device to be monitored is determined according to the initial status data, including: If the current moment is the data collection moment, determining the initial state data as the device state data; If the current moment is a moment between two data collection moments, the device state data is obtained by predicting based on the initial state data, or by calculating the initial state data through a device data model of a simulation system.

11. The method according to claim 10, characterized in that The real-time data of the device includes the posture data of the device to be monitored at the time of data collection and the motion data of the components included in the device to be monitored at the time of data collection; The position and posture data of the device to be monitored at the data collection time includes positioning data and posture data of the device to be monitored at the data collection time, the positioning data is obtained by a positioning device, and the posture data is obtained by a gyroscope; The motion data of the components included in the equipment to be monitored at the time of data collection are collected by sensors installed on the corresponding components.

12. A data processing device, characterized in that: The device comprises an acquisition unit, a rendering unit, a determination unit and a calculation unit: The acquisition unit is used to acquire scene model data of the scene to be monitored, and to acquire device status data of the device to be monitored in the scene to be monitored, wherein the device status data includes a device identification of the device to be monitored, position data of the device to be monitored at a current moment, and a motion state of a joint included in the device to be monitored at the current moment; The rendering unit is used to render a virtual scene of the scene to be monitored according to the scene model data; The determining unit is used to determine the instantiation object of the device to be monitored according to the device identification of the device to be monitored, wherein the instantiation object is generated by a first component description file, and the first component description file is used to indicate the components included in the device to be monitored and the joints connecting the components; The calculation unit is used to calculate the world transformation matrix corresponding to the device to be monitored according to the posture data of the device to be monitored; The rendering unit is further used to render the instantiated object in the virtual scene through the world transformation matrix; The calculation unit is further used to calculate the local transformation matrix of the joint according to the motion state of the joint included in the device to be monitored at the current moment; The determination unit is also used to determine the motion state of the virtual joint corresponding to the joint in the instantiated object at the current moment based on the local transformation matrix, and to determine the motion state of the component model connected to the virtual joint in the instantiated object at the current moment based on the local transformation matrix.

13. A data processing device, characterized in that: The device comprises an acquisition unit, a conversion unit, a determination unit and a sending unit: The acquisition unit is used to acquire the real-time data of the device to be monitored collected at the data collection time, and to acquire the first component description file of the device to be monitored; The conversion unit is used to convert the format of the device real-time data based on the first component description file to obtain initial state data; The determining unit is used to determine the device state data of the device to be monitored according to the initial state data, wherein the device state data includes the device identification of the device to be monitored, the position and posture data of the device to be monitored at a current moment, and the motion state of the joints included in the device to be monitored at the current moment; The sending unit is used to send the device status data of the device to be monitored to the device display terminal.

14. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1 to 11 according to the instructions in the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, which, when executed by a processor, enables the processor to perform the method according to any one of claims 1 to 11.

Images