Intelligent control application method and system based on digital twin model, and storage medium

Abstract

The application belongs to the technical field of digital twinning, and particularly relates to an intelligent control application method and system based on a digital twinning model and a storage medium, the method comprising the following steps: acquiring a three-dimensional coordinate point group about an internal environment of a factory and a video image about an internal production device of the factory, and generating a three-dimensional model of the internal environment of the factory and a three-dimensional model of the internal production device of the factory; collecting an operation parameter of the internal production device of the factory; displaying the three-dimensional model of the internal environment of the factory, and simultaneously displaying the three-dimensional model of the internal production device of the factory and the operation parameter of the internal production device of the factory in correspondence, and remotely regulating and controlling an operation state of the internal production device of the factory in the case that the operation parameter of the internal production device of the factory is abnormal. The three-dimensional model can be simply and efficiently generated through the application.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital twinning, and particularly relates to an intelligent control application method and system based on a digital twinning model and a storage medium. BACKGROUND

[0002] Digital twinning refers to creating a highly simulated virtual model for a physical entity in a digital manner, and jointly displaying the virtual model and data collected from the physical entity through a sensor. With the increasingly fast industrial intelligentization, establishing a digital twinning model of a factory can more intuitively monitor the production situation of the factory, and facilitate intelligent regulation and control of the production situation of the factory.

[0003] The application with the publication number CN115270642A discloses a plant factory intelligent management system based on digital twinning, which comprises a perception and execution layer for real-time sensing, monitoring and regulation of each element of a plant factory physical scene, a network transmission layer for providing an intelligent network transmission environment, adopting a multi-path routing algorithm for high-speed transmission and accurate distribution of data according to different functions and different subtasks, realizing two-way communication between a plant factory virtual model and an entity scene through setting different software interfaces, and dynamically mapping monitoring data in real time. The twinning function layer is used for establishing a plant factory digital twinning model of virtual-real mapping, and adopting a deep learning algorithm based on the digital twinning model to make intelligent regulation and control decisions of energy equipment and control equipment, different plant growth prediction and plant disease identification. The application with the publication number CN117390715A provides a digital twinning system based on a fracturing well factory operation line, which designs a well site in a digital manner through a digital well site design platform. Compared with a traditional two-dimensional design drawing, the system makes expression more intuitive through planning and design of a three-dimensional virtual entity, stores data of the virtual entity arranged in the virtual well site in an equipment database in an engineering construction budget system, calculates the required construction cost and construction time according to a bill of materials, and realizes overall monitoring of the fracturing well factory operation line through a digital twinning operation and maintenance system combined with the three-dimensional digital design drawing of the well site. However, the above-mentioned application does not explain how to establish a three-dimensional model. SUMMARY

[0004] The application obtains a three-dimensional coordinate point group about the factory internal environment and a video image about the factory internal production equipment, thereby generating a three-dimensional model of the factory internal environment and a three-dimensional model of the factory internal production equipment, and collecting an operation parameter of the factory internal production equipment, and displaying the three-dimensional model of the factory internal environment, and correspondingly displaying the three-dimensional model of the factory internal production equipment and the operation parameter of the factory internal production equipment, and remotely regulating the operation state of the factory internal production equipment in the case that the operation parameter of the factory internal production equipment is abnormal.

[0005] In order to achieve the above-mentioned application purposes, the application provides an intelligent control application method based on a digital twin model, mainly including the following steps:

[0006] S1, a scanning module is set, a three-dimensional coordinate point group about the factory internal environment is obtained through the scanning module, a photography module is set, a video image about the factory internal production equipment is collected using the photography module, and a control module is set, the control module generates a three-dimensional model of the factory internal environment according to the three-dimensional coordinate point group, and establishes a three-dimensional model of the factory internal production equipment based on the video image;

[0007] S2, a sensor module is set, an operation parameter of the factory internal production equipment is collected by relying on the sensor module, and a communication module is set, the sensor module transmits the operation parameter of the factory internal production equipment to the control module through the communication module;

[0008] S3, the control module displays the three-dimensional model of the factory internal environment, and correspondingly displays the three-dimensional model of the factory internal production equipment and the operation parameter of the factory internal production equipment, and the control module also analyzes and processes the operation parameter of the factory internal production equipment, and remotely regulates the operation state of the factory internal production equipment in the case that the operation parameter of the factory internal production equipment is abnormal.

[0009] As a preferred technical scheme of the application, the step of obtaining a three-dimensional coordinate point group about the factory internal environment through the scanning module includes using the scanning module to perform laser scanning on the upper part and the lower part of the factory internal environment, the front part and the rear part of the factory internal environment, and the left part and the right part of the factory internal environment, and simultaneously collecting the three-dimensional coordinate point group obtained through the scanning.

[0010] As a preferred technical scheme of the application, the control module generates a three-dimensional model of the factory internal environment according to the three-dimensional coordinate point group, including the following steps:

[0011] S111, in the three-dimensional coordinate point group, taking each three-dimensional coordinate point as a center coordinate point, determining three-dimensional coordinate points near the center coordinate point, and selecting two three-dimensional coordinate points from the near three-dimensional coordinate points to establish a triangular face with the center coordinate point each time, and calculating a feature quantity of the triangular face;

[0012] S112, calculating an average value of the several feature quantities corresponding to each three-dimensional coordinate point to obtain an average feature quantity corresponding to each three-dimensional coordinate point, dividing the three-dimensional coordinate points with the same corresponding average feature quantity into the same group, and determining the three-dimensional coordinate point group with the largest number of three-dimensional coordinate points as the three-dimensional coordinate point group of the upper and lower parts of the factory internal environment, and determining the remaining three-dimensional coordinate point groups as the three-dimensional coordinate point groups of the front and rear parts of the factory internal environment, and the left and right parts of the factory internal environment;

[0013] S113, in the three-dimensional coordinate point group of the upper and lower parts of the factory internal environment, determining the three-dimensional coordinate point group with a larger number of three-dimensional coordinate points as the three-dimensional coordinate point group of the upper part of the factory internal environment, and determining the three-dimensional coordinate point group with a smaller number of three-dimensional coordinate points as the three-dimensional coordinate point group of the lower part of the factory internal environment, and establishing an upper part model of the factory internal environment and a lower part model of the factory internal environment in the three-dimensional coordinate system respectively;

[0014] S114, calculating an average value of the average feature quantities corresponding to the three-dimensional coordinate point group of the upper part of the factory internal environment to obtain a reference feature quantity, and calculating an average value of the average feature quantities corresponding to the three-dimensional coordinate point groups of the front and rear parts of the factory internal environment and the left and right parts of the factory internal environment to obtain a calculation feature quantity respectively, and establishing a front and rear part model of the factory internal environment and a left and right part model of the factory internal environment in the three-dimensional coordinate system according to a mathematical relationship between the calculation feature quantity and the reference feature quantity.

[0015] As a preferred technical scheme of the present application, the three-dimensional model of the factory internal production equipment is established based on the video image, including the following steps:

[0016] S121, obtaining all pictures in the video image, and selecting a plurality of landmark points in the first picture, and recording a landmark point set of the first picture;

[0017] S122, determining whether all pictures in the video image have been processed, if yes, ending all steps, and if no, continuing the next step;

[0018] S123, acquire the next picture, determine the number of landmark points corresponding to the landmark points in the previous picture in the next picture, judge whether the number of landmark points is greater than the pre-set landmark point number threshold, in the case of the number of landmark points greater than the landmark point number threshold, record the landmark point set of the next picture, jump to S122, in the case of the number of landmark points less than or equal to the landmark point number threshold, continue S124;

[0019] S124, add new landmark points in the next picture, and divide all landmark points in the next picture into an old landmark point set containing only old landmark points and a new landmark point set containing new landmark points, record the old landmark point set and the new landmark point set of the next picture, and jump to S122.

[0020] As a preferred technical solution of the present application, the step of adding new landmark points in the picture includes: selecting a plurality of intermediate landmark points in the picture, calculating the interval value between the intermediate landmark points and each landmark point in the picture, accumulating different interval values to obtain the total interval value corresponding to the intermediate landmark points, and adding the intermediate landmark points corresponding to the largest total interval value as new landmark points to the picture.

[0021] As a preferred technical solution of the present application, the step of adding new landmark points in the picture is repeated before the number of all landmark points in the picture is greater than the landmark point number threshold.

[0022] As a preferred technical solution of the present application, the landmark point number threshold is set according to the complexity of the shape of the production equipment inside the factory. The higher the complexity of the shape of the production equipment inside the factory, the larger the landmark point number threshold is set.

[0023] As a preferred technical solution of the present application, the three-dimensional model of the production equipment inside the factory is established based on the video image, further comprising the following steps:

[0024] S125, regarding each picture in the video image, in the case of recording the landmark point set of the picture, performing triangulation processing on all landmark points in the landmark point set, calculating the sum of the areas of all triangular faces formed to obtain a total area value, and in the case of recording the old landmark point set and the new landmark point set, performing triangulation processing on all landmark points in the old landmark point set and the new landmark point set respectively, calculating the sum of the areas of all triangular faces formed respectively to obtain a first area value and a second area value, and calculating the average of the first area value and the second area value to obtain a total area value;

[0025] S126, select the picture in the video image corresponding to the maximum total area value, in the case of recording the landmark point set of the picture, use all the landmark points in the landmark point set to establish a three-dimensional model of the internal production equipment of the factory, in the case of recording the old landmark point set and the new landmark point set of the picture, use all the landmark points in the old landmark point set and the new landmark point set to establish a first model and a second model respectively, and further combine the first model and the second model into the three-dimensional model of the internal production equipment of the factory.

[0026] The application further provides an intelligent control application system based on a digital twin model, which comprises the following modules:

[0027] A scanning module is configured to acquire a three-dimensional coordinate point group about the internal environment of the factory.

[0028] A photography module is configured to collect video images about the internal production equipment of the factory.

[0029] A sensor module is configured to collect operation parameters of the internal production equipment of the factory.

[0030] A communication module is configured to transmit the operation parameters of the internal production equipment of the factory to the control module.

[0031] The control module is configured to generate a three-dimensional model of the internal environment of the factory according to the three-dimensional coordinate point group, and establish a three-dimensional model of the internal production equipment of the factory based on the video images; simultaneously configured to display the three-dimensional model of the internal environment of the factory, and correspondingly display the three-dimensional model of the internal production equipment of the factory and the operation parameters of the internal production equipment of the factory, and analyze and process the operation parameters of the internal production equipment of the factory, and remotely control the operation state of the internal production equipment of the factory in the case that the operation parameters of the internal production equipment of the factory are abnormal.

[0032] The application further provides a storage medium, which stores program instructions, wherein the program instructions control a device where the storage medium is located to execute the method in any one of the above.

[0033] Compared with the prior art, the application has at least the following advantages:

[0034] In the application, firstly, the three-dimensional coordinate point group about the factory internal environment is acquired through the scanning module, at the same time, the video image about the factory internal production equipment is collected using the photography module, and the three-dimensional model of the factory internal environment is generated according to the three-dimensional coordinate point group through the control module, the three-dimensional model of the factory internal production equipment is established according to the video image, secondly, the running parameters of the factory internal production equipment are collected by the sensor module, the running parameters of the factory internal production equipment are transmitted to the control module through the communication module, finally, the three-dimensional model of the factory internal environment is displayed through the control module, at the same time, the three-dimensional model of the factory internal production equipment and the running parameters of the factory internal production equipment are displayed correspondingly, and in the case that the running parameters of the factory internal production equipment are abnormal, the running state of the factory internal production equipment is remotely controlled through the control module. Through the application, not only the production situation of the factory can be monitored more intuitively and conveniently, but also the production equipment can be remotely controlled when necessary, and a simple, efficient and certain model precision generation three-dimensional model method can be provided. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The step flow chart of the intelligent control application method based on the digital twin model in the application;

[0036] Figure 2 The composition structure diagram of the intelligent control application system based on the digital twin model in the application;

[0037] Figure 3 An illustrative diagram in the application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application, and are not used to limit the application.

[0039] It can be understood that the terms "first", "second", etc. used in the present application can be used herein to describe various elements, but unless specifically stated, these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first xx script can be referred to as the second xx script, and similarly, the second xx script can be referred to as the first xx script.

[0040] The application provides an intelligent control application method based on a digital twin model as shown in Figure 1 The method is mainly realized by performing the following steps:

[0041] S1, set a scanning module, obtain a three-dimensional coordinate point group about the factory internal environment through the scanning module, set a photography module at the same time, collect video images about the factory internal production equipment using the photography module, and set a control module, the control module generates a three-dimensional model of the factory internal environment according to the three-dimensional coordinate point group, and also establishes a three-dimensional model of the factory internal production equipment based on the video images;

[0042] S2, set a sensor module, collect the running parameters of the factory internal production equipment by relying on the sensor module, and set a communication module, the sensor module transmits the running parameters of the factory internal production equipment to the control module through the communication module;

[0043] S3, the control module displays the three-dimensional model of the factory internal environment, and correspondingly displays the three-dimensional model of the factory internal production equipment and the running parameters of the factory internal production equipment, and the control module also analyzes and processes the running parameters of the factory internal production equipment, and remotely controls the running state of the factory internal production equipment in the case that the running parameters of the factory internal production equipment are abnormal.

[0044] Specifically, first, obtain a three-dimensional coordinate point group about the factory internal environment through the scanning module, collect video images about the factory internal production equipment using the photography module, and the control module generates a three-dimensional model of the factory internal environment according to the three-dimensional coordinate point group, and generates a three-dimensional model of the factory internal production equipment according to the video images. The specific generation method of the three-dimensional model will be described in detail below. Secondly, collect the running parameters of the factory internal production equipment by relying on the sensor module, and transmit the running parameters of the factory internal production equipment to the control module through the communication module. Finally, the control module displays the digital twin model of the factory, that is, displays the three-dimensional model of the factory internal environment, and correspondingly displays the three-dimensional model of the factory internal production equipment and the running parameters of the factory internal production equipment. The control module compares the running parameters of the factory internal production equipment with the normal range of the running parameters set in advance. If the running parameters exceed the normal range of the running parameters, it is determined that the running parameters of the factory internal production equipment are abnormal, that is, the running state of the factory internal production equipment is abnormal. The control module can remotely control the factory internal production equipment to stop running or change the running mode. Through the above method, based on the digital twin model of the factory, the production situation of the factory internal production equipment can be more intuitively monitored, and the running state of the factory internal production equipment can be remotely controlled when necessary.

[0045] Furthermore, the step of obtaining a group of three-dimensional coordinate points about the factory's internal environment through the scanning module includes using the scanning module to perform laser scanning on the upper and lower parts, the front and rear parts, and the left and right parts of the factory's internal environment, while collecting the group of three-dimensional coordinate points obtained through the scanning.

[0046] Specifically, in order to obtain a group of three-dimensional coordinate points about the factory's internal environment, the same scanning density is used to scan the upper and lower parts, the front and rear parts, and the left and right parts of the factory's internal environment using laser scanning. The upper part of the factory's internal environment can be understood as the roof, the lower part as the ground, and the front and rear, as well as the left and right parts, can be understood as the walls. In general, the roof, the ground, and the walls are all rectangular, and the lengths of the two sides corresponding to the roof are greater than the lengths of the sides corresponding to the height of the walls.

[0047] Furthermore, the control module generates a 3D model of the factory's internal environment based on the 3D coordinate point group, including the following steps:

[0048] S111. In a group of three-dimensional coordinate points, each three-dimensional coordinate point is taken as the center coordinate point. The three-dimensional coordinate points near the center coordinate point are determined. Each time, two three-dimensional coordinate points are selected from the nearby three-dimensional coordinate points and together with the center coordinate point to establish a triangular face. The characteristic quantities of the triangular face are calculated.

[0049] S112. Calculate the average value of several feature quantities corresponding to each three-dimensional coordinate point to obtain the average feature quantity corresponding to each three-dimensional coordinate point. Divide the three-dimensional coordinate points with the same average feature quantity into the same group. Determine the three-dimensional coordinate point group with the most three-dimensional coordinate points as the three-dimensional coordinate point group of the upper and lower parts of the factory internal environment. Determine the remaining three-dimensional coordinate point groups as the three-dimensional coordinate point groups of the front and rear parts of the factory internal environment, and the three-dimensional coordinate point groups of the left and right parts of the factory internal environment.

[0050] S113. In the three-dimensional coordinate point groups of the upper and lower parts of the factory's internal environment, the three-dimensional coordinate point group with more three-dimensional coordinate points is determined to be the three-dimensional coordinate point group of the upper part of the factory's internal environment, while the three-dimensional coordinate point group with fewer three-dimensional coordinate points is determined to be the three-dimensional coordinate point group of the lower part of the factory's internal environment. And establish the upper model and the lower model of the factory's internal environment in the three-dimensional coordinate system respectively.

[0051] S114, average values of average characteristic quantities corresponding to the group of three-dimensional coordinate points of the upper part of the internal environment of the factory are calculated to obtain a reference characteristic quantity, and average values of average characteristic quantities corresponding to the group of three-dimensional coordinate points of the front and back parts of the internal environment of the factory, and the left and right parts of the internal environment of the factory are calculated to obtain a calculation characteristic quantity respectively, and according to the mathematical relationship between the calculation characteristic quantity and the reference characteristic quantity, the front and back models of the internal environment of the factory, and the left and right models of the internal environment of the factory are established in the three-dimensional coordinate system respectively.

[0052] Specifically, in S111, each three-dimensional coordinate point in the three-dimensional coordinate point group is taken as a center coordinate point, a plurality of three-dimensional coordinate points near the center coordinate point are determined, two three-dimensional coordinate points are selected from the plurality of three-dimensional coordinate points near the center coordinate point each time to establish a triangular face together with the center coordinate point, and a characteristic quantity of the triangular face is calculated. It should be noted that the same center coordinate point can correspond to a plurality of triangular faces. The characteristic quantity can be a vector perpendicular to the triangular face. In S112, the average of the plurality of characteristic quantities corresponding to each three-dimensional coordinate point is calculated to obtain the average characteristic quantity corresponding to each three-dimensional coordinate point. The values in the average characteristic quantity also need to be normalized, such as being processed into a unit vector, so that the three-dimensional coordinate points corresponding to the same average characteristic quantity can be easily divided into the same group. For example, the three-dimensional coordinate points of the upper part of the factory internal environment are all divided into the same three-dimensional coordinate point group. Subsequently, the three-dimensional coordinate point group with the largest number of three-dimensional coordinate points is determined as the three-dimensional coordinate point group of the upper and lower parts of the factory internal environment, and the remaining three-dimensional coordinate point groups are determined as the three-dimensional coordinate point groups of the front and rear parts of the factory internal environment, and the left and right parts of the factory internal environment. This is because the area of the roof and the floor is much larger than the area of the wall, and the number of three-dimensional coordinate points is also much more. In S113, since production equipment is placed on the ground, the production equipment needs to be avoided during laser scanning. Therefore, the number of three-dimensional coordinate points corresponding to the roof is more than the number of three-dimensional coordinate points corresponding to the ground. In the three-dimensional coordinate point group of the upper and lower parts of the factory internal environment, the three-dimensional coordinate point group with the largest number of three-dimensional coordinate points is determined as the three-dimensional coordinate point group of the upper part of the factory internal environment, and the three-dimensional coordinate point group with the smallest number of three-dimensional coordinate points is determined as the three-dimensional coordinate point group of the lower part of the factory internal environment. Therefore, the upper model of the factory internal environment and the lower model of the factory internal environment can be established in the three-dimensional coordinate system, respectively. In S114, the average of the average characteristic quantities corresponding to the three-dimensional coordinate point group of the upper part of the factory internal environment is calculated to obtain a reference characteristic quantity. The average of the average characteristic quantities corresponding to the three-dimensional coordinate point groups of the front and rear parts of the factory internal environment and the left and right parts of the factory internal environment is also calculated to obtain different calculation characteristic quantities. Then, the front and rear models of the factory internal environment and the left and right models of the factory internal environment can be established in the three-dimensional coordinate system according to the mathematical relationship between the calculation characteristic quantities and the reference characteristic quantity. For reference, as shown in Figure 3 the three-dimensional models of the roof and the floor are parallel to the XOY plane, and the three-dimensional models of the walls are perpendicular to the three-dimensional models of the roof and the floor, the reference characteristic quantity is and the calculation characteristic quantities are and and the reference characteristic quantity is The perpendicular relationship between the two planes can be established by corresponding three-dimensional coordinate points.

[0053] Further, the three-dimensional model of the production equipment inside the factory is established based on the video images, including the following steps:

[0054] S121, all pictures in the video images are acquired, and a plurality of landmark points are selected in the first picture, and a landmark point set of the first picture is recorded;

[0055] S122, it is judged whether all pictures in the video images have been processed, if yes, all steps are ended, and if no, the next step is continued;

[0056] S123, the next picture is acquired, and the number of landmark points corresponding to the landmark points in the previous picture is determined in the next picture, it is judged whether the number of landmark points is greater than a preset landmark point number threshold, if the number of landmark points is greater than the landmark point number threshold, a landmark point set of the next picture is recorded, and S122 is jumped to, and if the number of landmark points is less than or equal to the landmark point number threshold, S124 is continued;

[0057] S124, a new landmark point is added in the next picture, and all landmark points in the next picture are divided into an old landmark point set containing only old landmark points and a new landmark point set containing new landmark points, the old landmark point set and the new landmark point set of the next picture are recorded, and S122 is jumped to.

[0058] Further, the step of adding a new landmark point in the picture includes: a plurality of intermediate landmark points are selected in the picture, interval values between the intermediate landmark points and each landmark point in the picture are calculated, different interval values are accumulated to obtain total interval values corresponding to the intermediate landmark points, and the intermediate landmark point corresponding to the largest total interval value is added to the picture as a new landmark point.

[0059] Further, the step of adding a new landmark point in the picture is repeated before the number of all landmark points in the picture is greater than the landmark point number threshold.

[0060] Further, the landmark point number threshold is set according to the complexity of the shape of the production equipment inside the factory, the higher the complexity of the shape of the production equipment inside the factory, the larger the landmark point number threshold is set.

[0061] Specifically, in S121, all pictures in the video image are acquired, and a plurality of landmark points are selected in the first picture, and a landmark point set of the first picture is recorded, wherein the landmark points can depict the shape of the object in the picture, in S122, it is determined whether all pictures in the video image have been processed, and the processing here refers to recording the landmark points in the picture, if yes, the whole step is ended, if no, the next step is continued, in S123, the next picture is acquired, and the number of landmark points corresponding to the landmark points in the previous picture is calculated in the picture, the number of landmark points is compared with the landmark point number threshold value, if the number of landmark points is greater than the landmark point number threshold value, the landmark point set of the picture is recorded, and S122 is jumped to continue to execute, if the number of landmark points is less than or equal to the landmark point number threshold value, S124 is continued to execute, in S124, new landmark points are added in the next picture, at this time, the new landmark points are added because the object in the picture changes or the shooting angle of the picture changes, so that the number of original landmark points is reduced too much, and the specific adding method will be introduced below, the next picture here refers to the picture acquired in S123, all landmark points in the picture are divided into an old landmark point set containing only old landmark points and a new landmark point set containing new landmark points, the new landmark point set contains new landmark points in addition to old landmark points, the old landmark point set and the new landmark point set of the picture are recorded, and S122 is jumped to continue to execute.

[0062] When the new landmark points are added in the picture, the step of adding the new landmark points needs to be repeated all the time, one new landmark point is added each time, until the number of all landmark points in the picture is greater than the landmark point number threshold value, the step of adding the new landmark points is that, first, a plurality of intermediate landmark points are selected in the picture, second, the interval values between the intermediate landmark points and each original landmark point in the picture are calculated, the different interval values are accumulated to obtain the total interval value corresponding to the intermediate landmark points, and finally, the intermediate landmark points corresponding to the largest total interval value are added to the picture as the new landmark points, each time the new landmark points are added, the intermediate landmark points can be reselected in the picture, in addition, the landmark point number threshold value is related to the complexity of the shape of the production equipment in the factory, and the higher the complexity of the shape of the production equipment in the factory is, the larger the landmark point number threshold value is set.

[0063] Further, the three-dimensional model of the production equipment in the factory is established based on the video image, and further includes the following steps:

[0064] S125、about each picture in the video image, in the case of recording is the landmark point set of the picture, all landmark points in the landmark point set are subjected to the triangulation processing, the sum of the areas of all triangular faces formed is calculated to obtain the total area value, and in the case of recording is the old landmark point set and the new landmark point set, all landmark points in the old landmark point set and the new landmark point set are subjected to the triangulation processing respectively, the sum of the areas of all triangular faces formed is calculated respectively to obtain the first area value and the second area value, and the average of the first area value and the second area value is calculated to obtain the total area value;

[0065] S126、select the picture in the video image corresponding to the largest total area value, in the case of recording is the landmark point set of the picture, all landmark points in the landmark point set are used to establish the three-dimensional model of the production equipment inside the factory, in the case of recording is the old landmark point set and the new landmark point set of the picture, all landmark points in the old landmark point set and the new landmark point set are used to establish the first model and the second model respectively, and the first model and the second model are merged into the three-dimensional model of the production equipment inside the factory.

[0066] Specifically, after the processes described in S121 to S124, in S125, considering each image in the video image, if the recorded data is a set of marker points, triangulation is performed on all marker points in the set, and the sum of the areas of all the resulting triangular faces is calculated to obtain the total area value. If the recorded data consists of an old set of marker points and a new set of marker points, triangulation is performed on all marker points in both sets, and the sum of the areas of all the resulting triangular faces is calculated to obtain a first area value and a second area value. Then, the average of the first and second area values ​​is calculated to obtain the total area value. In S126, the image in the video image corresponding to the largest total area value is selected. This is done because a larger total area value indicates a better shooting angle for the image; for example, the area calculated from a frontal view of the triangular face is larger. The area calculated from a triangular facet viewed from an oblique angle is larger than that calculated from a triangular facet. If the recorded data is a set of marker points from the selected image, a 3D model of the factory's internal production equipment is built using all the marker points in the set. If the recorded data is an old set of marker points and a new set of marker points from the selected image, a first model and a second model are built using all the marker points in the old and new sets, respectively. The first model and the second model are then merged into a 3D model of the factory's internal production equipment. This improves the accuracy of the generated 3D model. Before building the 3D model, it is necessary to calculate the 3D coordinates of the marker points in the image. This can be achieved using existing technology and will not be elaborated further. When merging the first model and the second model, the first model can be used as a reference. Based on the marker points in the second model that are the same as those in the first model, the second model is merged into the first model.

[0067] According to another aspect of the embodiments of the present invention, reference is made to... Figure 2 As shown, an intelligent control application system based on a digital twin model is also provided, including a scanning module 100, a photography module 200, a sensor module 300, a communication module 400, and a control module 500. The functions of each module are as follows:

[0068] The scanning module 100 is used to acquire a group of three-dimensional coordinate points about the internal environment of the factory.

[0069] Photography module 200 is used to collect video images of production equipment inside the factory;

[0070] Sensor module 300 is used to collect operating parameters of production equipment inside the factory;

[0071] The communication module 400 is used to transmit the operating parameters of the production equipment inside the factory to the control module 500.

[0072] The control module 500 is configured to generate a three-dimensional model of the internal environment of the factory according to the three-dimensional coordinate point group, and establish a three-dimensional model of the internal production equipment of the factory based on the video image; simultaneously configured to display the three-dimensional model of the internal environment of the factory, and correspondingly display the three-dimensional model of the internal production equipment of the factory and the running parameters of the internal production equipment of the factory, and analyze and process the running parameters of the internal production equipment of the factory, and remotely control the running state of the internal production equipment of the factory in the case that the running parameters of the internal production equipment of the factory are abnormal.

[0073] According to another aspect of the embodiments of the present application, a storage medium is also provided, which stores program instructions, wherein the program instructions control the device where the storage medium is located to execute the method of any one of the above embodiments when the program instructions are executed.

[0074] It should be understood that, although each step in the flowchart of each embodiment of the present application is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least a part of the steps in each embodiment can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least a part of other steps or sub-steps or stages of other steps.

[0075] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a non-volatile computer readable storage medium, and the program can include the processes of the above-mentioned embodiment methods when executed. Any reference to memory, storage, database, or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] Any combination of the technical features of the above-mentioned embodiments can be combined, and for the sake of brevity, not all possible combinations of the technical features in the above-mentioned embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0077] The above-mentioned embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as limiting the scope of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

[0078] The above-mentioned embodiments are only the preferred embodiments of the present application, and do not limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for intelligent control applications based on digital twin models, characterized in that, The steps include the following: S1. Set up a scanning module to acquire a group of three-dimensional coordinate points about the internal environment of the factory. At the same time, set up a photography module to collect video images about the production equipment inside the factory. Set up a control module to generate a three-dimensional model of the internal environment of the factory based on the group of three-dimensional coordinate points, and also to build a three-dimensional model of the production equipment inside the factory based on the video images. S2. Set up a sensor module to collect the operating parameters of the production equipment inside the factory, and set up a communication module to transmit the operating parameters of the production equipment inside the factory to the control module through the communication module. S3. The control module displays a three-dimensional model of the factory's internal environment, as well as a three-dimensional model of the production equipment inside the factory and its corresponding operating parameters. The control module also analyzes and processes the operating parameters of the production equipment inside the factory. If the operating parameters of the production equipment inside the factory become abnormal, the control module remotely adjusts the operating status of the production equipment inside the factory. Building a 3D model of the factory's internal production equipment based on the video images includes the following steps: S121, acquiring all images in the video images, and selecting several marker points in the first image to record the marker point set of the first image; S122, determining whether all images in the video images have been processed; if yes, ending all steps; otherwise, continuing to the next step; S123, acquiring the next image, and determining the number of marker points in the next image corresponding to the marker points in the previous image, determining whether the number of marker points is greater than a preset marker point number threshold; if the number of marker points is greater than the marker point number threshold, recording the next image. The set of marker points in the image is set, and the process jumps to S122. If the number of marker points is less than or equal to the marker point number threshold, the process continues to S124. In S124, new marker points are added to the next image, and all marker points in the next image are divided into an old marker point set containing only the old marker points and a new marker point set containing the new marker points. The old marker point set and the new marker point set of the next image are recorded, and the process jumps to S122. Here, a marker point is a point that can depict the shape of an object in the image. The marker point number threshold is set according to the complexity of the shape of the production equipment inside the factory. The higher the complexity of the shape of the production equipment inside the factory, the larger the marker point number threshold is set. The process of establishing a three-dimensional model of the factory's internal production equipment based on the video images further includes the following steps: S125. For each image in the video images, if the recorded set of marker points is a set of marker points, triangulation is performed on all marker points in the set of marker points, and the sum of the areas of all the triangular faces formed is calculated to obtain the total area value. If the recorded set of old marker points and set of new marker points are both set of marker points, triangulation is performed on all marker points in the old set of marker points and set of new marker points respectively, and the sum of the areas of all the triangular faces formed is calculated to obtain the first area value and the second area value. The average of the first area value and the second area value is also calculated to obtain the total area value. S126. Select the image from the video image corresponding to the largest total area value. If the recorded image is a set of marker points, use all marker points in the set to build a three-dimensional model of the factory's internal production equipment. If the recorded image is an old set of marker points and a new set of marker points, use all marker points in the old set of marker points and the new set of marker points to build a first model and a second model respectively. Then, merge the first model and the second model into a three-dimensional model of the factory's internal production equipment. When merging the first model and the second model, use the first model as a reference and merge the second model into the first model based on the marker points in the second model that are the same as the marker points in the first model. The step of obtaining a group of three-dimensional coordinate points about the factory's internal environment through the scanning module includes using the scanning module to perform laser scanning on the upper and lower parts, the front and rear parts, and the left and right parts of the factory's internal environment, while collecting the group of three-dimensional coordinate points obtained through the scanning. The steps for adding a new marker to an image include: selecting several intermediate markers in the image, calculating the interval between the intermediate marker and each of the original markers in the image, accumulating the different interval values ​​to obtain the total interval value corresponding to the intermediate marker, and adding the intermediate marker corresponding to the largest total interval value as a new marker to the image.

2. The method according to claim 1, characterized in that, The control module generates a three-dimensional model of the factory's internal environment based on the three-dimensional coordinate point group, including the following steps: S111. In the group of three-dimensional coordinate points, each three-dimensional coordinate point is taken as the center coordinate point. The three-dimensional coordinate points near the center coordinate point are determined. Each time, two three-dimensional coordinate points are selected from the nearby three-dimensional coordinate points and together with the center coordinate point to establish a triangular face. The characteristic quantities of the triangular face are calculated. S112. Calculate the average value of several feature quantities corresponding to each three-dimensional coordinate point to obtain the average feature quantity corresponding to each three-dimensional coordinate point. Divide the three-dimensional coordinate points with the same average feature quantity into the same group. Determine the three-dimensional coordinate point group with the most three-dimensional coordinate points as the three-dimensional coordinate point group of the upper and lower parts of the factory internal environment. Determine the remaining three-dimensional coordinate point groups as the three-dimensional coordinate point groups of the front and rear parts of the factory internal environment, and the three-dimensional coordinate point groups of the left and right parts of the factory internal environment. S113. In the three-dimensional coordinate point groups of the upper and lower parts of the factory's internal environment, the three-dimensional coordinate point group with more three-dimensional coordinate points is determined to be the three-dimensional coordinate point group of the upper part of the factory's internal environment, while the three-dimensional coordinate point group with fewer three-dimensional coordinate points is determined to be the three-dimensional coordinate point group of the lower part of the factory's internal environment. And establish the upper model and the lower model of the factory's internal environment in the three-dimensional coordinate system respectively. S114. Calculate the average value of the average feature quantity corresponding to the three-dimensional coordinate point group at the upper part of the factory's internal environment to obtain the reference feature quantity. At the same time, calculate the average value of the average feature quantity corresponding to the three-dimensional coordinate point group at the front and rear parts of the factory's internal environment, as well as the left and right parts of the factory's internal environment, to obtain the calculated feature quantity. Based on the mathematical relationship between the calculated feature quantity and the reference feature quantity, establish the front and rear models of the factory's internal environment, as well as the left and right models of the factory's internal environment, in the three-dimensional coordinate system.

3. The method according to claim 1, characterized in that, Before the total number of markers in the image exceeds the marker count threshold, repeat the process of adding new markers to the image.

4. An intelligent control application system based on a digital twin model, used to implement the method described in any one of claims 1-3, characterized in that, Includes the following modules: The scanning module is used to acquire a group of three-dimensional coordinate points about the factory's internal environment; The photography module is used to collect video images of production equipment inside the factory; The sensor module is used to collect operating parameters of production equipment inside the factory; The communication module is used to transmit the operating parameters of the production equipment inside the factory to the control module; The control module is used to generate a 3D model of the factory's internal environment based on a group of 3D coordinate points, and to build a 3D model of the factory's internal production equipment based on video images. It is also used to display the 3D model of the factory's internal environment, as well as the 3D model of the factory's internal production equipment and the corresponding display of the operating parameters of the factory's internal production equipment. Furthermore, it is used to analyze and process the operating parameters of the factory's internal production equipment, and to remotely control the operating status of the factory's internal production equipment when the operating parameters of the factory's internal production equipment are abnormal.

5. A storage medium, characterized in that, The storage medium stores program instructions, wherein when the program instructions are executed, they control the device where the storage medium is located to perform the method described in any one of claims 1-3.

Images