Driving method and device of digital twin model of fully mechanized mining working face

Abstract

The present application proposes a driving method and device for a digital twin model of a comprehensive mining working face, wherein the method includes: jointly calibrating the cameras in the comprehensive mining working face to obtain the calibration results corresponding to each camera; based on the calibration results corresponding to each camera, fusing the images collected by multiple cameras in real time to obtain a fused image; processing the fused image to obtain the action data corresponding to the working face equipment where the action occurs; based on the action data, driving the digital twin model corresponding to the working face equipment. The working face image can be collected in real time by the camera, and then the action data of the working face can be determined in real time and accurately, so as to realize the real-time and accurate driving of the digital twin model of the working face equipment, and ensure the consistency between the digital twin model corresponding to the working face equipment and the working face equipment.

Description

Technical Field

[0001] The present application relates to the field of coal mine technology, and in particular to a driving method for a digital twin model of a fully mechanized mining face. Background Art

[0002] With the continuous development of science and technology, digital twin technology, with its precise simulation and real-time monitoring characteristics, provides mining companies with more efficient and safer operation and management methods. Digital twin models can improve the safety and production efficiency of coal mining faces through digital means, and promote the development of the coal mining industry towards a safer, smarter and more sustainable direction. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of this application is to propose a driving method for the digital twin model of a comprehensive mining working face, so as to collect working face images in real time through a camera, and then determine the action data of the working face in real time and accurately, so as to realize real-time and accurate driving of the digital twin model of the working face equipment, and ensure the consistency between the digital twin model corresponding to the working face equipment and the working face equipment.

[0005] The second purpose of this application is to propose a driving device for the digital twin model of a comprehensive mining working face.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a driving method for a digital twin model of a fully mechanized mining face, comprising:

[0010] Perform joint calibration on the cameras in the fully mechanized mining face to obtain the calibration results corresponding to each camera;

[0011] Based on the calibration results corresponding to each camera, the images collected by multiple cameras are fused in real time to obtain a fused image;

[0012] Processing the fused image to obtain action data corresponding to the working surface equipment where the action occurs;

[0013] Based on the action data, the digital twin model corresponding to the work surface equipment is driven.

[0014] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a driving device for a digital twin model of a fully mechanized mining face, comprising:

[0015] The calibration module is used to jointly calibrate the cameras in the fully mechanized mining face to obtain the calibration results corresponding to each camera;

[0016] A fusion module is used to fuse images collected by multiple cameras in real time based on the calibration results corresponding to each camera to obtain a fused image;

[0017] A processing module, used for processing the fused image to obtain action data corresponding to the working surface equipment where the action occurs;

[0018] A driving module is used to drive the digital twin model corresponding to the working surface equipment based on the action data.

[0019] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including:

[0020] A processor, and a memory communicatively connected to the processor;

[0021] The memory stores computer-executable instructions;

[0022] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the embodiment of the first aspect.

[0023] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect embodiment.

[0024] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, a computer program, which implements the method described in the first aspect when executed by a processor.

[0025] The driving method and device of the digital twin model of the comprehensive mining working face provided in the present application jointly calibrates the cameras in the comprehensive mining working face to obtain the calibration results corresponding to each camera, and based on the calibration results corresponding to each camera, fuses the images collected by multiple cameras in real time to obtain a fused image, and then processes the fused image to obtain the action data corresponding to the working face equipment where the action occurs, and finally drives the digital twin model corresponding to the working face equipment based on the action data. Thus, the working face image can be collected in real time by the camera, and the action data of the working face can be determined in real time and accurately, so as to realize the real-time and accurate driving of the digital twin model of the working face equipment, and ensure the consistency between the digital twin model corresponding to the working face equipment and the working face equipment.

[0026] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A schematic flow chart of a driving method for a digital twin model of a fully mechanized mining face provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of a fully mechanized mining face provided in an embodiment of the present application;

[0030] Figure 3 A schematic flow chart of another method for driving a digital twin model of a fully mechanized mining face provided in an embodiment of the present application; and

[0031] Figure 4 A schematic diagram of the structure of a driving device for a digital twin model of a fully mechanized mining face provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0033] The following describes the driving method and device of the digital twin model of the comprehensive mining working face in an embodiment of the present application with reference to the accompanying drawings.

[0034] Figure 1 A flow chart of a driving method for a digital twin model of a fully mechanized mining face provided in an embodiment of the present application.

[0035] like Figure 1 As shown, the driving method of the digital twin model of the fully mechanized mining face includes the following steps:

[0036] Step 101, jointly calibrate the cameras in the comprehensive mining working face to obtain the calibration results corresponding to each camera.

[0037] The camera can be installed on a hydraulic support or a coal mining machine, but the present disclosure does not limit this.

[0038] Specifically, the cameras in the fully mechanized mining face are jointly calibrated to obtain the calibration results corresponding to each camera, which may include the following steps:

[0039] (1) Taking the first camera C1 installed at the head of the working surface as the origin of the global coordinate system, the positions of all cameras on the working surface in the global coordinate system are solved.

[0040] (2) The bracket number plate of the hydraulic bracket top beam hoisting is used as a reference marker, and the bracket number plate at the overlapping position of the camera C2 and camera C1 field of view is selected. The side length of the bracket number plate in the camera C1 video is identified and detected, and compared with the actual side length of the bracket number plate, and the conversion relationship between the center coordinate system of the number plate and the coordinate system of camera C1 is calculated.

[0041] It should be noted that the support license plate of the hydraulic support top beam is used as a reference marker. The four corner points of the license plate extracted from the image correspond to the four points assumed in the world coordinate system to form a 3D-2D point pair. According to the view geometry theory, the transformation matrix can be expressed as:

[0042]

[0043] Where H represents the transformation matrix, f x , f y represents the normalized focal length of the camera, u0, v0 represent the center coordinates of the image pixel, s represents the scale factor, (t x , t y , t z ) represents the translation vector, and R represents the rotation vector.

[0044] Here, we get the positions of the four corner points of the license plate in the camera C1 coordinate system and the camera C2 coordinate system. Then we calculate the positions of the center point of the license plate in the camera C1 coordinate system and the camera C2 coordinate system, and finally solve the translation and rotation matrices from the camera C2 coordinate system to the camera C1 coordinate system by using the singular value decomposition method.

[0045] (3) By analogy with step (2), solve the transformation relationship between the center coordinate system of the license plate and the coordinate system of camera C2;

[0046] (4) According to step (2) and step (3), the vectors of the four corner points of the bracket license plate in the two camera coordinates are solved, and the conversion relationship between the camera C2 coordinate system and the camera C1 coordinate system is jointly solved;

[0047] (5) Referring to step (1) and step (4), solve the transformation relationship between the coordinate systems of two adjacent cameras;

[0048] (6) Based on the coordinate transformation obtained in step (5), solve the transformation relationship between each camera coordinate system and the global coordinate system of camera C1.

[0049] Step 102: Based on the calibration result corresponding to each camera, images collected by multiple cameras are fused in real time to obtain a fused image.

[0050] It should be noted that, since one working surface camera can only capture working surface conditions within a certain range, the images captured by multiple working surface cameras can be integrated to monitor the overall conditions of the working surface.

[0051] In some embodiments, the data captured by the camera may be collected based on a preset sampling frequency.

[0052] Step 103: Process the fused image to obtain action data corresponding to the working surface equipment that has taken action.

[0053] In some embodiments, the fused image can be processed to determine the position information, travel direction, travel speed and position information of the front drum of the coal shearer. The coal shearer is a working face device that takes action, and the position information, travel direction, travel speed and position information of the front drum of the coal shearer are the action data of the coal shearer.

[0054] Optionally, the position information, travel direction, travel speed and position information of the front drum of the coal mining machine can be determined by the following steps:

[0055] (1) Recognize the fused image and determine the center position of the front drum rocker arm and the shearer body;

[0056] (2) Determine the frame number of the hydraulic support corresponding to the center position of the coal mining machine body and the frame number of the hydraulic support corresponding to the front drum rocker arm of the coal mining machine, wherein the frame number of the hydraulic support corresponding to the center position is the position information of the coal mining machine;

[0057] (3) According to the displacement of the center position of the coal mining machine body between two adjacent frames of fused images, the corresponding walking direction and walking speed of the coal mining machine are determined;

[0058] (4) According to the frame number of the hydraulic support corresponding to the rocker arm of the front drum of the coal mining machine and the structural relationship between the rocker arm of the front drum of the coal mining machine and the cutting drum, the front drum contour line and the relative position relationship between the front drum contour line and the hydraulic support are determined, wherein the relative position relationship can be used to indicate the position information of the front drum of the coal mining machine.

[0059] The relative position relationship may be the frame number of the hydraulic support corresponding to the front roller contour line.

[0060] In some embodiments, the fused image can be processed to determine the first hydraulic support in the guard plate retraction area, the retraction action of the first hydraulic support, the first angle between the edge identification lines of each level of guard plates corresponding to the first hydraulic support after the retraction action is completed, and the second angle between the edge identification line of the first level of guard plates corresponding to the first hydraulic support and the lower edge identification line of the top beam. Among them, the first hydraulic support is the working surface equipment that takes action, and the retraction action of the first hydraulic support, the first angle and the second angle are the action data corresponding to the first hydraulic support.

[0061] Optionally, the retracting action of the first hydraulic support, the first angle and the second angle may be obtained by the following steps:

[0062] (1) According to the relative position relationship, determine the first hydraulic support affected by the front roller contour line.

[0063] Among them, the first hydraulic support can be a hydraulic support corresponding to the contour line of the front roller.

[0064] (2) The fused image is identified to obtain the edge identification line corresponding to each level of the side guard plate of the first hydraulic support and the lower edge identification line of the top beam of the first hydraulic support.

[0065] (3) Based on the calibration results corresponding to each camera, the edge identification lines corresponding to each level of the side guard plate and the edge identification lines under the top beam are converted to the camera coordinate system respectively.

[0066] Specifically, based on the actual length of each level of the multi-level side guard plates and the camera internal parameters, the edge identification line corresponding to each level of the side guard plates and the edge identification line under the top beam can be converted to the camera coordinate system respectively.

[0067] (4) In the camera coordinate system, the retraction action of the first hydraulic support is determined according to the change in the angle between the edge identification lines of each level of the guard plate corresponding to the first hydraulic support and the change in the angle between the edge identification line of the first level of the guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam;

[0068] (5) After the retraction action is completed, determine the first angle between the edge identification lines of each level of the guard plate corresponding to the first hydraulic support, and the second angle between the edge identification line of the first level of the guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam.

[0069] In some embodiments, the fused image can be processed to determine the second hydraulic support in the guard plate opening area, the opening action of the second hydraulic support, the third angle between the edge identification lines corresponding to each level of guard plates of the second hydraulic support after the opening action is completed, and the fourth angle between the edge identification line of the first level of guard plates corresponding to the second hydraulic support and the lower edge identification line of the top beam. Among them, the second hydraulic support is the working surface equipment that takes action, and the opening action of the second hydraulic support, the third angle and the fourth angle are the action data corresponding to the second hydraulic support.

[0070] Optionally, the opening action, the third angle and the fourth angle of the second hydraulic support may be obtained by the following steps:

[0071] (1) Determine that the hydraulic support located between the center position of the coal mining machine body and the first hydraulic support is the second hydraulic support;

[0072] (2) Recognizing the fused image to obtain edge identification lines corresponding to each level of the side guard plate of the second hydraulic support and the lower edge identification line of the top beam of the second hydraulic support;

[0073] (3) Based on the calibration results corresponding to each camera, the edge identification lines corresponding to each level of the side guard plate of the second hydraulic support and the lower edge identification line of the top beam of the second hydraulic support are respectively converted to the camera coordinate system;

[0074] (4) In the camera coordinate system, the opening action of the second hydraulic support is determined according to the change in the angle between the edge identification lines of each level of the guard plate corresponding to the second hydraulic support and the change in the angle between the edge identification line of the first level of the guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam;

[0075] (5) After the opening action is completed, determine the third angle between the edge identification lines of each level of the guard plate corresponding to the second hydraulic support, and the fourth angle between the edge identification line of the first level of the guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam.

[0076] In some embodiments, the fused image may be processed to determine the third hydraulic support in the frame moving area, the frame moving action of the third hydraulic support, and the first distance between the third hydraulic support and the scraper conveyor when the frame moving action stops. The third hydraulic support is the working surface equipment that takes action, and the frame moving action of the third hydraulic support and the first distance are the action data corresponding to the third hydraulic support.

[0077] Optionally, the third hydraulic support of the frame moving device, the frame moving action of the third hydraulic support, and the first distance between the third hydraulic support and the scraper conveyor when the frame moving action stops may be determined by the following steps:

[0078] (1) A first number of adjacent hydraulic supports starting from a position at which the distance from the center position of the coal mining machine is a first preset distance may be determined as the third hydraulic support, wherein the third hydraulic support is located behind the coal mining machine in the direction of travel;

[0079] (2) Identifying the fused image to determine the first straight line connected to the coal retaining plate of the scraper conveyor when the frame moving action stops, and the first edge line of the front end of the third hydraulic support base;

[0080] (3) based on the calibration results corresponding to each camera, convert the first straight line and the first edge line into the camera coordinate system respectively;

[0081] (4) determining a horizontal distance between the first straight line and the first edge line in camera coordinates;

[0082] (5) when it is detected that the horizontal distance between the first straight line and the first edge line begins to decrease, determining that the third hydraulic support starts to move;

[0083] (6) When it is detected that the horizontal distance between the first straight line and the first edge line remains unchanged, the third hydraulic support is determined to stop moving, and the horizontal distance between the first straight line and the first edge line when the moving of the support stops is determined as the first distance.

[0084] In some embodiments, the horizontal distance between the first straight line formed by the coal retaining plate of the scraper conveyor and the first edge line at the front end of the base of the third hydraulic support can be detected in real time. When the detected horizontal distance becomes smaller, the height of the digital twin model corresponding to the third hydraulic support is driven to be reduced by 20 cm; the length of the push hydraulic cylinder is calculated according to the structural relationship of the push jack and the horizontal distance, and sent to the digital twin server of the working face, the digital twin model corresponding to the scraper conveyor is set to be fixed, and the digital twin model body corresponding to the third hydraulic support is driven to move forward; when it is detected that the horizontal distance between the first straight line and the second edge line is unchanged, the digital twin model body corresponding to the third hydraulic support is driven to stop moving forward and rise by 20 cm. In this way, the digital twin model corresponding to the third hydraulic support can be updated in real time and accurately to ensure the consistency between the digital twin model corresponding to the third hydraulic support and the third hydraulic support in the actual working face.

[0085] In some embodiments, the fused image may be processed to determine the fourth hydraulic support for pushing the scraper conveyor, the pushing action of the fourth hydraulic support, and the second distance between the fourth hydraulic support and the scraper conveyor when the pushing action stops. The scraper conveyor is the working surface equipment where the action occurs, and the pushing action of the fourth hydraulic support and the second distance are the action data corresponding to the scraper conveyor.

[0086] Specifically, the fourth hydraulic support for pushing the scraper conveyor, the pushing action of the fourth hydraulic support, and the second distance between the fourth hydraulic support and the scraper conveyor when the pushing action stops can be determined by the following steps:

[0087] (1) A second number of adjacent hydraulic supports starting from a position at which the distance from the center position of the coal mining machine is a second preset distance may be determined as the fourth hydraulic support, wherein the fourth hydraulic support is located behind the coal mining machine in the direction of travel;

[0088] (2) Identifying the fused image to determine a second straight line formed by the coal retaining plates of the scraper conveyor when the pushing action stops, and a second edge line at the front end of the base of the fourth hydraulic support;

[0089] (3) Based on the calibration results corresponding to each camera, the second straight line and the second edge line are respectively converted into the camera coordinate system;

[0090] (4) determining a horizontal distance between the second straight line and the second edge line in the camera coordinates;

[0091] (5) when it is detected that the horizontal distance between the second straight line and the second edge line begins to increase, determining that the fourth hydraulic support begins to move;

[0092] (6) When it is detected that the horizontal distance between the second straight line and the second edge line remains unchanged, the fourth hydraulic support is determined to stop moving, and the horizontal distance between the second straight line and the second edge line when the movement stops is determined as the second distance.

[0093] Figure 2 A schematic diagram of a fully mechanized mining face provided in the embodiment of the present application. Figure 2 As shown, D hb One or more hydraulic supports corresponding to the area are the first hydraulic supports affected by the front roller contour line, D hh One or more hydraulic supports corresponding to the area are the second hydraulic supports with the guard plate opened, D y is the first preset distance, D ij One or more hydraulic supports corresponding to the area are the third hydraulic supports of the moving frame, D t is the second preset distance, D ty The hydraulic support corresponding to the area is the fourth hydraulic support of the push scraper conveyor.

[0094] Step 104: Drive the digital twin model corresponding to the working surface equipment based on the action data.

[0095] When the working face equipment in action is a coal mining machine, the digital twin model corresponding to the coal mining machine is driven to move based on the position information, walking direction, walking speed and position information of the front drum of the coal mining machine.

[0096] When the working face equipment in action is the first hydraulic support for retracting the guard plate, after determining the retraction action of the first hydraulic support, the lengths of the driving cylinders at each level are determined according to the change in the angle between the edge identification lines of each level of guard plate corresponding to the first hydraulic support, the change in the angle between the edge identification line of the first level of guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam, and the guard plate structure of the first hydraulic support. Then, based on the lengths of the driving cylinders at each level, the digital twin model of the first hydraulic support is driven to retract the guard plate. Finally, based on the first angle and the second angle, the digital twin model of the first hydraulic support is driven to stop retracting the guard plate.

[0097] When the working face equipment that takes action is the second hydraulic support for opening the guard plate, after determining the opening action of the second hydraulic support, the lengths of each level of driving cylinders are determined according to the change in the angle between the edge identification lines of each level of guard plate corresponding to the second hydraulic support, and the change in the angle between the edge identification line of the first level of guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam. Then, based on the lengths of each level of driving cylinders, the digital twin model of the second hydraulic support is driven to open the guard plate. Finally, according to the third angle and the fourth angle, the digital twin model of the second hydraulic support is driven to stop opening the guard plate.

[0098] When the working face equipment that takes action is the third hydraulic support that moves the frame, after determining that the third hydraulic support starts to move the frame, the digital twin model of the third hydraulic support is driven to lower the preset height; according to the horizontal distance between the first straight line and the first edge line, and the structural relationship of the push jack of the hydraulic support, the length of the first push hydraulic cylinder is determined, and then based on the length of the first push hydraulic cylinder, the digital twin model of the third hydraulic support is driven to move the frame; finally, based on the first distance, the digital twin model of the third hydraulic support is driven to stop moving the frame and raise the preset height.

[0099] When the working face equipment in action is the fourth hydraulic support of the scraper conveyor, after determining that the fourth hydraulic support starts to move, the length of the second pushing hydraulic cylinder is determined according to the horizontal distance between the second straight line and the second edge line, and the structural relationship of the pushing jack of the hydraulic support. Then, based on the length of the second pushing hydraulic cylinder, the digital twin model of the scraper conveyor is driven to move. Finally, based on the second distance, the digital twin model of the fourth hydraulic support is driven to stop moving.

[0100] In this embodiment, the cameras in the comprehensive mining working face are jointly calibrated to obtain the calibration results corresponding to each camera. Based on the calibration results corresponding to each camera, the images collected by multiple cameras are fused in real time to obtain a fused image. The fused image is then processed to obtain the action data corresponding to the working face equipment that takes action. Finally, based on the action data, the digital twin model corresponding to the working face equipment is driven. In this way, the working face image can be collected in real time by the camera, and the action data of the working face can be determined in real time and accurately, so as to realize the real-time and accurate driving of the digital twin model of the working face equipment, and ensure the consistency between the digital twin model corresponding to the working face equipment and the working face equipment.

[0101] Figure 3 A flow chart of a driving method for a digital twin model of a fully mechanized mining face provided in an embodiment of the present application.

[0102] like Figure 3 As shown, the driving method of the digital twin model of the fully mechanized mining face includes the following steps:

[0103] Step 301, jointly calibrate the cameras in the fully mechanized mining face to obtain the calibration results corresponding to each camera.

[0104] Step 302: Based on the calibration result corresponding to each camera, the images collected by multiple cameras are fused in real time to obtain a fused image.

[0105] Step 303, processing the fused image, determining the position information, walking direction, walking speed and position information of the front drum of the coal mining machine, and driving the digital twin model corresponding to the coal mining machine to walk.

[0106] Step 304, processing the fused image, determining the first hydraulic support in the guard plate retraction area, the retraction action of the first hydraulic support, the first angle between the edge identification lines of each level of guard plates corresponding to the first hydraulic support after the retraction action is completed, and the second angle between the edge identification line of the first level of guard plates corresponding to the first hydraulic support and the lower edge identification line of the top beam; according to the retraction action of the first hydraulic support, the first angle, the second angle, and the guard plate structure of the first hydraulic support, driving the digital twin model of the first hydraulic support to retract the guard plate.

[0107] Step 305, after the front drum of the coal mining machine cuts coal, the fused image is processed to determine the second hydraulic support in the guard plate opening area and the opening action of the second hydraulic support. After the opening action is completed, the third angle between the edge identification lines corresponding to each level of guard plates of the second hydraulic support and the fourth angle between the edge identification line of the first level of guard plates corresponding to the second hydraulic support and the lower edge identification line of the top beam are determined. According to the opening action of the second hydraulic support, the third angle, the fourth angle, and the guard plate structure of the second hydraulic support, the digital twin model of the second hydraulic support is driven to open the guard plate.

[0108] Step 306, process the fused image, determine the third hydraulic support in the frame moving area, the frame moving action of the third hydraulic support, and the first distance between the third hydraulic support and the scraper conveyor when the frame moving action stops, and drive the digital twin model of the third hydraulic support to move the frame according to the frame moving action of the third hydraulic support, the first distance and the structural relationship of the push jack of the hydraulic support.

[0109] Step 307, processing the fused image, determining the fourth hydraulic support in the area of ​​the scraper conveyor, the pushing action of the fourth hydraulic support, and the second distance between the fourth hydraulic support and the scraper conveyor when the pushing action stops, and driving the digital twin model of the fourth hydraulic support to move according to the pushing action of the fourth hydraulic support, the second distance and the structural relationship of the pushing jack of the hydraulic support.

[0110] In some embodiments, the horizontal distance between the second straight line formed by the coal baffles of the scraper conveyor and the second edge line at the front end of the base of the fourth hydraulic support can be detected in real time. When the horizontal distance between the second straight line and the second edge line increases, the length of the pushing hydraulic cylinder is calculated according to the structural relationship of the pushing jack and the horizontal distance between the second straight line and the second edge line, and sent to the digital twin server of the working face to drive the digital twin model corresponding to the fourth hydraulic support to remain stationary, and the digital twin model of the scraper conveyor moves forward as a whole. At the second distance, the fourth hydraulic support is driven to stop pushing, thereby completing the pushing action drive of the fourth hydraulic support.

[0111] In some embodiments, steps 302 to 307 are repeated. After completing the driving of one coal-cutting action, step 301 is repeated. After completing the camera position calibration, the digital twin model driving of the next coal-cutting action is performed.

[0112] In this embodiment, during the coal mining process, the working face equipment that takes action at each stage can be determined based on the coal mining action, and then the fused image can be analyzed to determine the action data corresponding to the working face equipment to drive the corresponding digital twin model. As a result, the working face equipment that should be detected based on the fused image at each stage can be determined in real time during the coal mining process, reducing the amount of data processed at each stage and improving the efficiency and timeliness of the corresponding digital twin model.

[0113] In order to implement the above-mentioned embodiments, the present application also proposes a driving device for the digital twin model of the comprehensive mining working face.

[0114] Figure 4 A schematic diagram of the structure of a driving device of a digital twin model of a fully mechanized mining face provided in an embodiment of the present application. Figure 4 As shown, the driving device of the digital twin model of the fully mechanized mining face includes:

[0115] The calibration module 401 is used to jointly calibrate the cameras in the fully mechanized mining face to obtain the calibration results corresponding to each camera;

[0116] A fusion module 402 is used to fuse images collected by multiple cameras in real time based on the calibration results corresponding to each camera to obtain a fused image;

[0117] The processing module 403 is used to process the fused image to obtain the action data corresponding to the working surface equipment where the action occurs;

[0118] The driving module 404 is used to drive the digital twin model corresponding to the working surface equipment based on the action data.

[0119] Furthermore, in a possible implementation of the embodiment of the present application, the processing module 403 is used to:

[0120] Process the fused image to determine the position information, travel direction, travel speed and position information of the front drum of the coal mining machine;

[0121] Processing the fused image, determining the first hydraulic support in the guard plate retraction area, the retraction action of the first hydraulic support, the first angle between the edge identification lines of each level of guard plates corresponding to the first hydraulic support after the retraction action is completed, and the second angle between the edge identification line of the first level of guard plates corresponding to the first hydraulic support and the lower edge identification line of the top beam;

[0122] Process the fused image to determine the second hydraulic support in the guard plate opening area, the opening action of the second hydraulic support, the third angle between the edge identification lines corresponding to each level of guard plates of the second hydraulic support after the opening action is completed, and the fourth angle between the edge identification line of the first level of guard plates corresponding to the second hydraulic support and the lower edge identification line of the top beam;

[0123] Processing the fused image to determine the third hydraulic support in the frame moving area, the frame moving action of the third hydraulic support, and the first distance between the third hydraulic support and the scraper conveyor when the frame moving action stops;

[0124] The fused image is processed to determine the fourth hydraulic support in the area of ​​the scraper conveyor, the pushing action of the fourth hydraulic support, and the second distance between the fourth hydraulic support and the scraper conveyor when the pushing action stops.

[0125] Furthermore, in a possible implementation of the embodiment of the present application, the processing module 403 is used to:

[0126] Identify the fused image and determine the center position of the front drum rocker arm and the shearer body;

[0127] Determine the frame number of the hydraulic support corresponding to the center position of the coal mining machine body and the frame number of the hydraulic support corresponding to the front drum rocker arm of the coal mining machine, wherein the frame number of the hydraulic support corresponding to the center position is the position information of the coal mining machine;

[0128] According to the displacement of the center position of the coal mining machine body between two adjacent fusion image frames, the corresponding walking direction and walking speed of the coal mining machine are determined;

[0129] According to the frame number of the hydraulic support corresponding to the front drum rocker arm of the coal mining machine and the structural relationship between the front drum rocker arm and the cutting drum of the coal mining machine, the front drum contour line and the relative position relationship between the front drum contour line and the hydraulic support are determined, wherein the relative position relationship is used to indicate the position information of the front drum of the coal mining machine.

[0130] Furthermore, in a possible implementation of the embodiment of the present application, the processing module 403 is used to:

[0131] Determine the first hydraulic support affected by the front roller contour line according to the relative position relationship;

[0132] Recognize the fused image to obtain the edge identification line corresponding to each level of the side guard plate of the first hydraulic support and the lower edge identification line of the top beam of the first hydraulic support;

[0133] Based on the calibration results corresponding to each camera, the edge identification line corresponding to each level of the side guard plate and the lower edge identification line of the top beam are converted to the camera coordinate system respectively;

[0134] In the camera coordinate system, the retraction action of the first hydraulic support is determined according to the change in the angle between the edge identification lines of each level of the guard plate corresponding to the first hydraulic support and the change in the angle between the edge identification line of the first level of the guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam;

[0135] After the retraction action is completed, determine the first angle between the edge identification lines of each level of guard plate corresponding to the first hydraulic support, and the second angle between the edge identification line of the first level guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam.

[0136] Furthermore, in a possible implementation of the embodiment of the present application, the driving module 404 is configured to:

[0137] After the retracting action of the first hydraulic support is determined, the lengths of the driving cylinders at each level are determined according to the change in the angle between the edge identification lines of each level of the guard plate corresponding to the first hydraulic support, the change in the angle between the edge identification line of the first level of the guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam, and the structure of the guard plate of the first hydraulic support;

[0138] Based on the lengths of the driving cylinders at each level, the digital twin model of the first hydraulic support is driven to retract the guard plate;

[0139] Based on the first angle and the second angle, the digital twin model of the first hydraulic support is driven to stop retracting the guard plate.

[0140] Furthermore, in a possible implementation of the embodiment of the present application, the processing module 403 is used to:

[0141] Determine that the hydraulic support located between the center position of the coal mining machine body and the first hydraulic support is the second hydraulic support;

[0142] Identify the fused image to obtain the edge identification line corresponding to each level of the side guard plate of the second hydraulic support and the lower edge identification line of the top beam of the second hydraulic support;

[0143] Based on the calibration results corresponding to each camera, the edge identification lines corresponding to each level of the side guard plate of the second hydraulic support and the lower edge identification line of the top beam of the second hydraulic support are respectively converted to the camera coordinate system;

[0144] In the camera coordinate system, the opening action of the second hydraulic support is determined according to the change in the angle between the edge identification lines of each level of the guard plate corresponding to the second hydraulic support and the change in the angle between the edge identification line of the first level of the guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam;

[0145] After the opening action is completed, the third angle between the edge identification lines of each level of guard plate corresponding to the second hydraulic support and the fourth angle between the edge identification line of the first level guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam are determined.

[0146] Furthermore, in a possible implementation of the embodiment of the present application, the driving module 404 is configured to:

[0147] After the opening action of the second hydraulic support is determined, the lengths of the driving cylinders at each level are determined according to the change in the angle between the edge identification lines of each level of the guard plate corresponding to the second hydraulic support and the change in the angle between the edge identification line of the first level of the guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam;

[0148] Based on the length of each level of driving cylinders, the digital twin model of the second hydraulic support is driven to open the side guard plate;

[0149] According to the third angle and the fourth angle, the digital twin model of the second hydraulic support is driven to stop opening the side guard plate.

[0150] Furthermore, in a possible implementation of the embodiment of the present application, the processing module 403 is used to:

[0151] Determine a first number of adjacent hydraulic supports starting from a position where the distance from the center position of the coal mining machine is a first preset distance as the third hydraulic support, wherein the third hydraulic support is located behind the traveling direction of the coal mining machine;

[0152] Identify the fused image to determine the first straight line formed by the coal retaining plates of the scraper conveyor and the first edge line of the front end of the third hydraulic support base;

[0153] Based on the calibration results corresponding to each camera, the first straight line and the first edge line are respectively converted into the camera coordinate system;

[0154] In camera coordinates, determining a horizontal distance between the first straight line and the first edge line;

[0155] When it is detected that the horizontal distance between the first straight line and the first edge line begins to decrease, determining that the third hydraulic support starts to move;

[0156] When it is detected that the horizontal distance between the first straight line and the first edge line is unchanged, it is determined that the third hydraulic support stops moving, and the horizontal distance between the first straight line and the first edge line when the moving of the support stops is determined as the first distance.

[0157] Furthermore, in a possible implementation of the embodiment of the present application, the driving module 404 is configured to:

[0158] After determining that the third hydraulic support starts to move, driving the digital twin model of the third hydraulic support to lower a preset height;

[0159] Determine the length of the first push hydraulic cylinder according to the horizontal distance between the first straight line and the first edge line and the structural relationship of the push jack of the hydraulic support;

[0160] Based on the length of the first push hydraulic cylinder, the digital twin model of the third hydraulic support is driven to move the support;

[0161] Based on the first distance, the digital twin model of the third hydraulic support is driven to stop moving the support and raise the preset height.

[0162] Furthermore, in a possible implementation of the embodiment of the present application, the processing module 403 is used to:

[0163] Determine a second number of adjacent hydraulic supports starting from a position at which the distance from the center position of the coal mining machine is a second preset distance as the fourth hydraulic support, wherein the fourth hydraulic support is located behind the traveling direction of the coal mining machine;

[0164] Identify the fused image to determine the second straight line formed by the coal retaining plates of the scraper conveyor when the pushing action stops, and the second edge line of the front end of the fourth hydraulic support base;

[0165] Based on the calibration results corresponding to each camera, the second straight line and the second edge line are respectively converted into the camera coordinate system;

[0166] In the camera coordinates, determining the horizontal distance between the second straight line and the second edge line;

[0167] When it is detected that the horizontal distance between the second straight line and the second edge line begins to increase, determining that the fourth hydraulic support begins to move;

[0168] When it is detected that the horizontal distance between the second straight line and the second edge line remains unchanged, it is determined that the fourth hydraulic support stops moving, and the horizontal distance between the second straight line and the second edge line when the movement stops is determined as the second distance.

[0169] Furthermore, in a possible implementation of the embodiment of the present application, the driving module 404 is configured to:

[0170] After determining that the fourth hydraulic support starts to move, determining the length of the second push hydraulic cylinder according to the horizontal distance between the second straight line and the second edge line and the structural relationship of the push jack of the hydraulic support;

[0171] Based on the length of the second hydraulic cylinder, the digital twin model of the scraper conveyor is driven to move;

[0172] Based on the second distance, the digital twin model of the fourth hydraulic support is driven to stop moving.

[0173] It should be noted that the above explanation of the embodiment of the driving method of the digital twin model of the comprehensive mining working face is also applicable to the driving device of the digital twin model of the comprehensive mining working face of this embodiment, and will not be repeated here.

[0174] In this embodiment, the cameras in the comprehensive mining working face are jointly calibrated to obtain the calibration results corresponding to each camera. Based on the calibration results corresponding to each camera, the images collected by multiple cameras are fused in real time to obtain a fused image. The fused image is then processed to obtain the action data corresponding to the working face equipment that takes action. Finally, based on the action data, the digital twin model corresponding to the working face equipment is driven. In this way, the working face image can be collected in real time by the camera, and the action data of the working face can be determined in real time and accurately, so as to realize the real-time and accurate driving of the digital twin model of the working face equipment, and ensure the consistency between the digital twin model corresponding to the working face equipment and the working face equipment.

[0175] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0176] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0177] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0178] In order to implement the above-mentioned embodiments, the present application also proposes a digital twin model driving system for a comprehensive mining working face based on visual scanning, which is characterized in that it includes a comprehensive mining working face camera, a video processing server and a digital twin server. The video processing server implements image processing related steps, such as step 103, and the digital twin server implements digital twin model driving related steps, such as step 104.

[0179] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0180] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0181] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0182] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0183] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0184] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0185] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0186] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0187] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0189] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A driving method for a digital twin model of a fully mechanized mining face, characterized in that: The following steps are involved: Perform joint calibration on the cameras in the fully mechanized mining face to obtain the calibration results corresponding to each camera; Based on the calibration results corresponding to each camera, the images collected by multiple cameras are fused in real time to obtain a fused image; Processing the fused image to obtain action data corresponding to the working surface equipment where the action occurs; Based on the action data, driving the digital twin model corresponding to the working surface equipment; The processing of the fused image to obtain action data corresponding to the working face equipment in action includes: processing the fused image to determine the position information, walking direction, walking speed and position information of the front drum of the coal mining machine; processing the fused image to determine the first hydraulic support in the guard plate retraction area, the retraction action of the first hydraulic support, the first angle between the edge identification lines of each level of guard plates corresponding to the first hydraulic support after the retraction action is completed, and the second angle between the edge identification line of the first level of guard plates corresponding to the first hydraulic support and the lower edge identification line of the top beam; The processing of the fused image to determine the position information, the travel direction, the travel speed of the coal mining machine and the position information of the front drum of the coal mining machine includes: Identify the fused image to determine the center position of the front drum rocker arm and the body of the coal mining machine; Determine the frame number of the hydraulic support corresponding to the center position of the coal mining machine body, and the frame number of the hydraulic support corresponding to the front drum rocker arm of the coal mining machine, wherein the frame number of the hydraulic support corresponding to the center position is the position information of the coal mining machine; Determine the corresponding walking direction and walking speed of the coal mining machine according to the displacement of the center position of the coal mining machine body between two adjacent frames of fused images; According to the frame number of the hydraulic support corresponding to the rocker arm of the front drum of the coal mining machine and the structural relationship between the rocker arm of the front drum of the coal mining machine and the cutting drum, the front drum contour line and the relative position relationship between the front drum contour line and the hydraulic support are determined, wherein the relative position relationship is used to indicate the position information of the front drum of the coal mining machine; The processing of the fused image to determine the first hydraulic support in the guard plate retraction area, the retraction action of the first hydraulic support, the first angle between the edge identification lines of each level of guard plates corresponding to the first hydraulic support after the retraction action is completed, and the second angle between the edge identification line of the first level of guard plates corresponding to the first hydraulic support and the lower edge identification line of the top beam, includes: Determine the first hydraulic support affected by the front roller contour line according to the relative position relationship; Identify the fused image to obtain edge identification lines corresponding to each level of the side guard plates of the first hydraulic support and a lower edge identification line of the top beam of the first hydraulic support; Based on the calibration results corresponding to each camera, the edge identification lines corresponding to each level of the side guard plate and the lower edge identification line of the top beam are respectively converted to the camera coordinate system; In the camera coordinate system, the retracting action of the first hydraulic support is determined according to the change in the angle between the edge identification lines of each level of the side guard plate corresponding to the first hydraulic support and the change in the angle between the edge identification line of the first level side guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam; After the retraction action is completed, it is determined that the first hydraulic support corresponds to a first angle between edge identification lines of each level of guard plates, and a second angle between the edge identification line of the first level of guard plates corresponding to the first hydraulic support and the lower edge identification line of the top beam.

2. The method according to claim 1, characterized in that The processing of the fused image to obtain action data corresponding to the working surface equipment that has taken action further includes at least one of the following: Processing the fused image, determining a second hydraulic support in the guard plate opening area, an opening action of the second hydraulic support, a third angle between edge identification lines corresponding to each level of guard plates of the second hydraulic support after the opening action is completed, and a fourth angle between an edge identification line of a first level of guard plates corresponding to the second hydraulic support and an edge identification line of a lower portion of a top beam; Processing the fused image to determine a third hydraulic support in a frame shifting area, a frame shifting action of the third hydraulic support, and a first distance between the third hydraulic support and the scraper conveyor when the frame shifting action stops; The fused image is processed to determine a fourth hydraulic support that pushes the scraper conveyor area, a pushing action of the fourth hydraulic support, and a second distance between the fourth hydraulic support and the scraper conveyor when the pushing action stops.

3. The method according to claim 2, characterized in that The driving of the digital twin model corresponding to the working surface equipment based on the action data includes: After determining the retracting action of the first hydraulic support, determining the length of each level of driving cylinder according to the change of the angle between the edge identification lines of each level of the guard plate corresponding to the first hydraulic support, the change of the angle between the edge identification line of the first level of the guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam, and the structure of the guard plate of the first hydraulic support; Based on the lengths of the driving cylinders at each level, the digital twin model of the first hydraulic support is driven to retract the guard plate; Based on the first angle and the second angle, the digital twin model of the first hydraulic support is driven to stop retracting the side guard plate.

4. The method according to claim 2, characterized in that: The processing of the fused image to determine the second hydraulic support in the guard plate opening area, the opening action of the second hydraulic support, the third angle between the edge identification lines corresponding to each level of guard plates of the second hydraulic support after the opening action is completed, and the fourth angle between the edge identification line of the first level of guard plates corresponding to the second hydraulic support and the lower edge identification line of the top beam includes: Determine the hydraulic support located between the center position of the coal mining machine body and the first hydraulic support as the second hydraulic support; Identify the fused image to obtain edge identification lines corresponding to each level of the side guard plate of the second hydraulic support and a lower edge identification line of the top beam of the second hydraulic support; Based on the calibration results corresponding to each camera, the edge identification lines corresponding to each level of the side guard plate of the second hydraulic support and the lower edge identification line of the top beam of the second hydraulic support are respectively converted to the camera coordinate system; In the camera coordinate system, the opening action of the second hydraulic support is determined according to the change in the angle between the edge identification lines of each level of the side guard plate corresponding to the second hydraulic support and the change in the angle between the edge identification line of the first level of the side guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam; After the opening action is completed, the third angle between the edge identification lines of each level of guard plate corresponding to the second hydraulic support and the fourth angle between the edge identification line of the first level guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam are determined.

5. The method according to claim 4, characterized in that The driving of the digital twin model corresponding to the working surface equipment based on the action data includes: After the opening action of the second hydraulic support is determined, the lengths of the driving cylinders at each level are determined according to the change in the angle between the edge identification lines of each level of the guard plate corresponding to the second hydraulic support and the change in the angle between the edge identification line of the first level of the guard plate corresponding to the second hydraulic support and the lower edge identification line of the top beam; Based on the lengths of the driving cylinders at each level, driving the digital twin model of the second hydraulic support to open the side guard plate; According to the third angle and the fourth angle, the digital twin model of the second hydraulic support is driven to stop opening the side guard plate.

6. The method according to claim 2, characterized in that The processing of the fused image to determine the third hydraulic support in the frame shifting area, the frame shifting action of the third hydraulic support, and the first distance between the third hydraulic support and the scraper conveyor when the frame shifting action stops includes: Determine a first number of adjacent hydraulic supports starting from a position at which the distance from the center position of the coal mining machine is a first preset distance as the third hydraulic supports, wherein the third hydraulic supports are located behind the coal mining machine in the traveling direction; Identify the fused image to determine a first straight line formed by the coal retaining plates of the scraper conveyor and a first edge line at the front end of the base of the third hydraulic support; Based on the calibration results corresponding to each camera, convert the first straight line and the first edge line into the camera coordinate system respectively; Under the camera coordinates, determining a horizontal distance between the first straight line and the first edge line; When it is detected that the horizontal distance between the first straight line and the first edge line begins to decrease, determining that the third hydraulic support starts to move; When it is detected that the horizontal distance between the first straight line and the first edge line remains unchanged, it is determined that the third hydraulic support stops moving, and the horizontal distance between the first straight line and the first edge line when the moving of the support stops is determined as the first distance.

7. The method according to claim 6, characterized in that The driving of the digital twin model corresponding to the working surface equipment based on the action data includes: After determining that the third hydraulic support starts to move, driving the digital twin model of the third hydraulic support to lower a preset height; Determine the length of the first push hydraulic cylinder according to the horizontal distance between the first straight line and the first edge line and the structural relationship of the push jack of the hydraulic support; Based on the length of the first push hydraulic cylinder, drive the digital twin model of the third hydraulic support to move the support; Based on the first distance, the digital twin model of the third hydraulic support is driven to stop moving the support and rise to a preset height.

8. The method according to claim 2, characterized in that: The processing of the fused image to determine the fourth hydraulic support pushing the scraper conveyor area, the pushing action of the fourth hydraulic support, and the second distance between the fourth hydraulic support and the scraper conveyor when the pushing action stops, includes: Determine a second number of adjacent hydraulic supports starting from a position at which the distance from the center position of the coal mining machine is a second preset distance as the fourth hydraulic support, wherein the fourth hydraulic support is located behind the traveling direction of the coal mining machine; Identify the fused image to determine a second straight line formed by the coal retaining plates of the scraper conveyor and a second edge line of the front end of the base of the fourth hydraulic support when the pushing action stops; Based on the calibration results corresponding to each camera, convert the second straight line and the second edge line into the camera coordinate system respectively; Under the camera coordinates, determining a horizontal distance between the second straight line and the second edge line; When it is detected that the horizontal distance between the second straight line and the second edge line begins to increase, determining that the fourth hydraulic support begins to move; When it is detected that the horizontal distance between the second straight line and the second edge line remains unchanged, it is determined that the fourth hydraulic support stops moving, and the horizontal distance between the second straight line and the second edge line when the movement stops is determined as the second distance.

9. The method according to claim 8, characterized in that The driving of the digital twin model corresponding to the working surface equipment based on the action data includes: After determining that the fourth hydraulic support starts to move, determining the length of the second push hydraulic cylinder according to the horizontal distance between the second straight line and the second edge line and the structural relationship of the push jack of the hydraulic support; Based on the length of the second push hydraulic cylinder, driving the digital twin model of the scraper conveyor to move; Based on the second distance, the digital twin model of the fourth hydraulic support is driven to stop moving.

10. A driving device for a digital twin model of a fully mechanized mining face, characterized in that: include: The calibration module is used to jointly calibrate the cameras in the fully mechanized mining face to obtain the calibration results corresponding to each camera; A fusion module is used to fuse images collected by multiple cameras in real time based on the calibration results corresponding to each camera to obtain a fused image; A processing module, used for processing the fused image to obtain action data corresponding to the working surface equipment where the action occurs; A driving module, used to drive the digital twin model corresponding to the working surface equipment based on the action data; The processing module is further used to process the fused image to determine the position information, walking direction, walking speed and position information of the front drum of the coal mining machine; process the fused image to determine the first hydraulic support in the guard plate retraction area, the retraction action of the first hydraulic support, the first angle between the edge identification lines of each level of guard plates corresponding to the first hydraulic support after the retraction action is completed, and the second angle between the edge identification line of the first level of guard plates corresponding to the first hydraulic support and the lower edge identification line of the top beam; The processing module is specifically used to identify the fused image and determine the center position of the front drum rocker arm and the coal mining machine body of the coal mining machine; Determine the frame number of the hydraulic support corresponding to the center position of the coal mining machine body, and the frame number of the hydraulic support corresponding to the front drum rocker arm of the coal mining machine, wherein the frame number of the hydraulic support corresponding to the center position is the position information of the coal mining machine; Determine the corresponding walking direction and walking speed of the coal mining machine according to the displacement of the center position of the coal mining machine body between two adjacent frames of fused images; According to the frame number of the hydraulic support corresponding to the rocker arm of the front drum of the coal mining machine and the structural relationship between the rocker arm of the front drum of the coal mining machine and the cutting drum, the front drum contour line and the relative position relationship between the front drum contour line and the hydraulic support are determined, wherein the relative position relationship is used to indicate the position information of the front drum of the coal mining machine; Determine the first hydraulic support affected by the front roller contour line according to the relative position relationship; Identify the fused image to obtain edge identification lines corresponding to each level of the side guard plates of the first hydraulic support and a lower edge identification line of the top beam of the first hydraulic support; Based on the calibration results corresponding to each camera, the edge identification lines corresponding to each level of the side guard plate and the lower edge identification line of the top beam are respectively converted to the camera coordinate system; In the camera coordinate system, the retracting action of the first hydraulic support is determined according to the change in the angle between the edge identification lines of each level of the side guard plate corresponding to the first hydraulic support and the change in the angle between the edge identification line of the first level side guard plate corresponding to the first hydraulic support and the lower edge identification line of the top beam; After the retraction action is completed, it is determined that the first hydraulic support corresponds to a first angle between edge identification lines of each level of guard plates, and a second angle between the edge identification line of the first level of guard plates corresponding to the first hydraulic support and the lower edge identification line of the top beam.

Images